Method for training diffusion model, and diffusion model-based image generation method
By introducing downsampling during the forward diffusion process and upsampling during the backward diffusion process of the diffusion model, the problems of high computational complexity and low training stability in generating high-resolution images by the diffusion model are solved, achieving efficient and direct generation of high-resolution images.
Patent Information
- Application Number
- PCT/CN2026/091217
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-08
- Filing Date
- 2026-04-17
- Publication Date
- 2026-11-12
AI Technical Summary
Current diffusion models have high computational complexity when generating high-resolution images, making it difficult to directly generate high-resolution images. Furthermore, multi-stage training strategies and post-processing modules result in low training stability and poor generalization ability.
By introducing downsampling during forward diffusion and upsampling during backward diffusion, noisy sample images with various resolutions and noise levels are constructed, and the diffusion model is trained so that it can directly output images of different resolutions.
It simplifies the model training process and resource consumption, improves training efficiency and generated image quality, avoids resolution fluctuation issues, and enables the direct generation of high-resolution images.
Smart Images

Figure CN2026091217_12112026_PF_FP_ABST
Abstract
Description
Training methods for diffusion models and image generation methods based on diffusion models Technical Field
[0001] One or more embodiments of this specification relate to the field of machine learning, and in particular to a method for training a diffusion model and an image generation method based on a diffusion model. Background Technology
[0002] As a powerful generative model, diffusion models are widely used in image and video generation. Typical diffusion models include the Dall·E model, the Imagen model, and the Stable Diffusion model. Current diffusion models require multiple steps of gradual denoising during inference, resulting in high computational complexity and making it difficult to directly generate high-resolution images.
[0003] To address this, several methods have been proposed to improve the diffusion model. For example, a multi-stage training strategy can be employed: first, a low-resolution diffusion model is trained, and then, through initializing model weights, diffusion models of different resolutions are trained sequentially from low to high. This approach is complex, time-consuming, and its training stability and model quality depend heavily on numerous tricks, resulting in typically low generalization ability. Alternatively, a super-resolution post-processing module can be added after the conventional diffusion model to further process the diffusion model's output, thereby obtaining a high-resolution image. However, this post-processing approach often fails to fully utilize the diffusion model's generative capabilities, and the post-processing module's results for different images are usually inconsistent, easily leading to significant resolution fluctuations across different video frames of the same video, resulting in poor video quality. Summary of the Invention
[0004] In view of the above, one or more embodiments of this specification provide the following technical solutions.
[0005] According to a first aspect of one or more embodiments of this specification, a training method for a diffusion model is proposed, comprising a forward diffusion process and a backward diffusion process, wherein: in the forward diffusion process, noise from real samples at levels 1 to T1 is added to a noiseless level 0 noisy sample image in order from time step 1 to T1, and downsampling is performed to obtain noisy sample images at levels 1 to T1 with progressively decreasing resolution, where T1 is a positive integer greater than 1; in the backward diffusion process, the diffusion model is trained using the noisy sample images at levels 1 to T1 until a stopping condition is met; wherein, for an integer i satisfying 1≤i≤T1, the noisy sample image at time step i and the level i resolution is input into the diffusion model to obtain the level i predicted sample noise output by the model, wherein the deviation between the level i-1 real sample noise and the level i-1 predicted sample noise obtained by upsampling the level i predicted sample noise is used to adjust the parameters of the diffusion model.
[0006] According to a second aspect of one or more embodiments of this specification, an image generation method based on a diffusion model is proposed. The method includes: in response to a received image generation instruction, invoking a diffusion model to perform T2 rounds of denoising processing on a T2-level denoised image of pure noise in the order of time steps from T2 to 1, so as to obtain a noise-free image, wherein the diffusion model is trained by the method described in the first aspect; wherein, for an integer j satisfying 1≤j≤T2, denoising processing is performed on the j-th level denoised image, including: inputting the j-th level denoised image at time step j and the j-th level resolution into the diffusion model to obtain the j-th level predicted noise output by the model, and determining a j-1 level denoised image at a j-1 level resolution by upsampling based on the j-th level denoised image and the j-th level predicted noise, wherein the 0-th level denoised image determined when j=1 is used as the noise-free image.
[0007] According to a third aspect of one or more embodiments of this specification, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in the first or second aspect by executing the executable instructions.
[0008] According to a fourth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first or second aspect.
[0009] According to a fifth aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the method as described in the first or second aspect.
[0010] As can be seen from the above embodiments, this specification improves the training and inference methods of conventional diffusion models. The conventional diffusion model training process consists of a forward diffusion process (constructing training samples by progressively adding noise, i.e., gradually converting the original data into noise) and a backward diffusion process (training the model by progressively denoising, i.e., gradually recovering the original data from the noise). This scheme adds downsampling processing to the conventional forward diffusion process, thereby constructing noisy sample images with different resolutions and noise levels. These downsampled noisy sample images are then used as samples in the backward diffusion process, combined with upsampling, to train the diffusion model. This enables the trained diffusion model to "directly output noisy images of different resolutions." Correspondingly, in the model inference stage, by inputting a low-resolution pure noise image into the diffusion model and specifying the corresponding time step, and performing corresponding upsampling processing during the model's progressive denoising process, a high-resolution noise-free image can be finally obtained from the model output.
[0011] Understandably, this solution integrates resolution-specific processing logic into the diffusion model by introducing downsampling during forward diffusion and upsampling during backward diffusion and inference. This allows the diffusion model to generate noise-free images from low to high resolution with only one complete training cycle, eliminating the need for multi-stage, multiple training sessions. This significantly simplifies the training process, reduces resource consumption, and improves training efficiency. Furthermore, downsampling reduces the computational load during training and inference (significantly lower at low resolution), accelerating training and inference. In the denoising process, progressive upsampling from time step T2 to T1 allows the model to better learn and recover high-resolution details, improving the quality of generated images and videos. Finally, the diffusion model itself generates high-resolution noise-free images in a single step, avoiding resolution fluctuations introduced by post-processing modules. Attached Figure Description
[0012] Figure 1 is a schematic diagram of the training method of conventional diffusion models in related technologies.
[0013] Figure 2 is a flowchart of a training method for a diffusion model provided in an exemplary embodiment.
[0014] Figure 3 is a schematic diagram of a training method for a diffusion model provided in an exemplary embodiment.
[0015] Figure 4 is a flowchart of an image generation method based on a diffusion model provided in an exemplary embodiment.
[0016] Figure 5 is a schematic diagram of the structure of a device provided in an exemplary embodiment.
[0017] Figure 6 is a block diagram of a training apparatus for a diffusion model provided in an exemplary embodiment.
[0018] Figure 7 is a block diagram of an image generation apparatus based on a diffusion model provided in an exemplary embodiment. Detailed Implementation
[0019] As a powerful generative model, diffusion models are widely used in image and video generation. Typical diffusion models include the Dall·E model, the Imagen model, and the Stable Diffusion model. Current diffusion models require multiple steps of gradual denoising during inference, resulting in high computational complexity and making it difficult to directly generate high-resolution images.
[0020] To address this, several methods have been proposed to improve the approach. For example, a multi-stage training strategy can be employed to train diffusion models. This involves first training a low-resolution diffusion model, and then, through initializing model weights, training diffusion models at different resolutions sequentially from low to high. However, this method is complex, time-consuming, and its training stability and model quality rely heavily on numerous tricks, resulting in typically low generalization ability. Specifically, during the training of any resolution model, the image at that resolution needs to be sequentially denoised and de-denoised. Taking the training process of a 1920×1080px diffusion model as an example, as shown in Figure 1, it specifically includes a forward diffusion process for adding noise and a backward diffusion process for denoising (or noise reduction). Each process comprises T (T being a positive integer greater than 1) time steps. In the forward diffusion process, noise is added sequentially from time step 1 to n; correspondingly, in the backward diffusion process, noise is removed sequentially from time step n to 1. The first rectangle represents a noise-free image with a resolution of 1920×1080px, before any noise has been added.
[0021] Alternatively, a super-resolution post-processing module could be added after the conventional diffusion model to further process the output of the diffusion model and obtain a high-resolution image. However, this type of post-processing often fails to fully utilize the generative capabilities of the diffusion model, and the processing results of the post-processing module for different images are often inconsistent, easily leading to large fluctuations in resolution between different video frames of the same video, resulting in poor video quality.
[0022] To address the aforementioned technical problems in related technologies, this specification improves upon the conventional training method of diffusion models by proposing a novel training method for diffusion models and a method for generating images based on the diffusion model trained using this method. The aim is to simplify the training and inference processes of the model by introducing downsampling during forward diffusion and upsampling during backward diffusion, thereby obtaining a diffusion model capable of directly inferring and outputting images of a specified resolution. This solution is described below with reference to the accompanying drawings and related embodiments.
[0023] The training process of the diffusion model described in the embodiments of this specification includes a forward diffusion process and a backward diffusion process. The forward diffusion process gradually transforms the original data (such as a noise-free original sample image X0) into pure noise X by progressively adding noise to it. T The noise addition at each step follows a Markov chain, meaning the current state depends only on the previous state. The forward diffusion process disrupts the original data distribution, gradually transforming a complex data distribution into a simple Gaussian distribution to facilitate subsequent reverse generation. The noise data generated in this process is used as samples to train the denoising model (i.e., the diffusion model) during the reverse diffusion process. The reverse diffusion process learns to progressively denoise from the noise X... T The original sample image X0 is reconstructed. This process uses a diffusion model based on a neural network to predict noise or mean, gradually corrects the data distribution, and finally generates a noise-free image.
[0024] The upsampling and downsampling processes described in the embodiments of this specification are not fundamentally different from upsampling and downsampling in related technologies. Downsampling is used to reduce computational complexity or extract more abstract features by reducing the size or resolution of the data. In deep learning, downsampling is typically used to reduce the spatial dimensions (such as height and width) of feature maps, while potentially increasing the number of channels. Conversely, upsampling is used to restore or generate higher-resolution outputs by increasing the size or resolution of the data, typically used to restore low-resolution feature maps to their original size or a larger size.
[0025] Figure 2 is a flowchart of a diffusion model training method provided in an exemplary embodiment, which can be applied to a model training system. As shown in Figure 2, the method includes the following steps 202-204.
[0026] Step 202: During the forward diffusion process, noise from the first to the first level (T1) real samples is added to the noiseless first level 0 sample image in the order of time steps from 1 to T1 and then downsampled to obtain the first to the first level (T1) noisy sample images with progressively lower resolution, where T1 is a positive integer greater than 1.
[0027] The forward diffusion process is used to construct training samples for the diffusion model by progressively adding noise. In the embodiments of this specification, starting from a noise-free image, corresponding noise is progressively added in order of time step t from 1 to T1 (i.e., from small to large). This noise-free image is called the level 0 noise-added sample image, and the added noise is called the real sample noise. Here, time step i (i.e., time step t = i, the same below, where 1 ≤ i ≤ T1) corresponds to the level i noise-added sample image with the level i resolution (i.e., the resolution level is level i). Further adding level i real sample noise to this image and downsampling it yields the next level (i.e., the level i+1 noise-added sample image). As can be seen from the basic principle of downsampling, the resolution of the level i noise-added sample data is higher than the resolution of the level i+1 noise-added sample data obtained by downsampling the level i noise-added sample data. Therefore, the resolution of the level 1 to T1 noise-added sample images obtained through the above noise addition and downsampling processes gradually decreases.
[0028] The embodiments in this specification can specify the resolution information of the corresponding image through the diffusion model at time step t. Therefore, the value of time step t corresponds to the resolution of the noisy sample image. A mapping relationship between time steps and resolution levels can be pre-created. For example, this mapping relationship can be seen in Table 1 below: Table 1: Mapping Relationship between Time Steps and Resolution Levels
[0029] As can be seen, there is a one-to-one correspondence between the time step t and the resolution in Table 1: the larger the value of the time step t, the higher the resolution level, and the lower the resolution of the corresponding image. It is understandable that the diffusion model trained according to the above mapping relationship will have the ability to generate noise-free images at the corresponding resolution level. The one-to-one correspondence between the time step t and the resolution is to ensure that the diffusion model can cover as many resolutions as possible, thereby enabling it to generate noise-free images at more resolutions. In fact, the time step t described in this scheme can also have a many-to-one relationship with the resolution, that is, multiple values of the time step t correspond to the same resolution. For example, time step t values of 1 to 3 all correspond to resolution level 0, and time step t value of 4 corresponds to resolution level 4. The diffusion model trained based on this mapping relationship will be able to generate noise-free images at resolution levels 1 and 4, which will not be elaborated further.
[0030] In one embodiment, when adding level 1 to level T1 real sample noise to a noiseless level 0 noisy sample image in sequence from time step 1 to T1 and then downsampling to obtain level 1 to T1 noisy sample images with progressively decreasing resolution, different methods can be adopted as needed. For example, level 1 to level T1 real sample noise can be added to a noiseless level 0 noisy sample image in sequence from time step 1 to T1, and each level of noise addition result can be downsampled to obtain a noisy sample image of the corresponding level. The level i noisy sample image to which noise is added should have the same resolution as the level i real sample noise added. Alternatively, level 0 noisy sample image can be downsampled in sequence from time step 1 to T1, and the corresponding level of real sample noise can be added to each downsampling result to obtain a noisy sample image of the corresponding level. The level i downsampling result should also have the same resolution as the level i real sample noise added.
[0031] During downsampling, downsampling operators are used to reduce image resolution by aggregating or discarding pixels. In one embodiment, common downsampling operators can be used, such as max pooling (which helps preserve significant features like image texture by taking the maximum value within a local window), average pooling (which effectively smooths background information by calculating the mean of a local window), strided convolution (where the kernel stride needs to be set to be greater than 1 to skip pixels for downsampling), and Gaussian pyramid (which generates multi-scale images by interleaving after layer-by-layer Gaussian blurring), etc., which will not be elaborated further. Preferably, the aforementioned downsampling operators can be linear or differentiable operators to facilitate gradient optimization of the diffusion model during backdiffusion.
[0032] In one embodiment, in addition to adding Gaussian noise, non-Gaussian noise such as impulse noise, alpha-stable noise, and colored noise can also be added. Alternatively, Gaussian-impulse mixed noise, Poisson noise, or quantization noise can also be added. The specific type of noise added can be flexibly selected based on the image characteristics, and this specification does not limit this. The following description uses Gaussian noise as an example.
[0033] In one embodiment, during the forward diffusion process, let's assume that the i-th level noisy sample image is x.t The downsampling operator is Down(x) t-1 ), where the (i+1)th level noisy sample image is shown in the following equation (1): q(x t |x t-1 )=N(x t Down(|x t-1 ),β t I) (1)
[0034] Where, β t These are noise scheduling parameters used to control the noise intensity at each step.
[0035] After the aforementioned noise addition and downsampling processes, the resulting noisy sample images of resolutions 1 to T1 can be used to construct training samples for training the diffusion model during backdiffusion. Each training sample can include a noisy sample image and the real sample noise added when generating that image. For example, for a noisy sample image obtained by adding i-th level real sample noise to an i-1 level noisy sample image and then downsampling, the i-th level noisy sample image and the i-th level real sample noise constitute a training sample.
[0036] Following Table 1, the process of obtaining the noisy sample images of levels 1 to T1 through forward diffusion can be seen in Figure 3. As shown in Figure 3, the resolution of the level 0 noisy sample image is 5120×2880px. By progressively adding noise and downsampling at time steps t from 1 to 7, the following noisy sample images are obtained: level 0 noisy sample image with a resolution of 3840×2160px, level 2 noisy sample image with a resolution of 2560×1440px, ..., level 7 noisy sample image with a resolution of 256×256px.
[0037] The foregoing embodiments only illustrate the noise addition and downsampling process of a noiseless Level 0 noise-added sample image. In practice, a large number of noiseless images can be pre-acquired as Level 0 noise-added sample images, and noise can be added to each image and downsampled in the manner described in the previous embodiments, thereby obtaining Level 1 to T1 noise-added sample images corresponding to each image. The sample image set composed of these images can be used for the subsequent back-diffusion process.
[0038] Step 204: During the back diffusion process, the diffusion model is trained using the noisy sample images of levels 1 to T1 until the stopping condition is met. For an integer i satisfying 1 ≤ i ≤ T1, the noisy sample image of level i at time step i and level i resolution is input into the diffusion model to obtain the predicted sample noise of level i output by the model. The deviation between the real sample noise of level i-1 and the predicted sample noise of level i-1 obtained by upsampling the predicted sample noise of level i is used to adjust the parameters of the diffusion model.
[0039] In one embodiment, the diffusion model described in this specification can be built using a DIT (Diffusion Transformer) network, where Transformer blocks can be used at each scale to capture global and local feature information. Alternatively, a multi-scale neural network such as U-net can be used, in which case the model employs an encoder-decoder structure, progressively downsampling input features in the encoder and progressively upsampling features in the decoder to achieve resolution conversion. Furthermore, skip connections can be established between the encoder and decoder to fuse features from different scales, more accurately preserve spatial information, and improve the accuracy of the final output image.
[0040] In one embodiment, the number of level 0 noisy sample images is multiple. A set of sample images is formed by adding noise to each level 0 noisy sample image and then downsampling it to obtain level 1 to T1 noisy sample images. Based on this, when determining the level i-th noisy sample image at level i resolution during the back-diffusion process, at least one level i-th noisy sample image can be randomly selected from all level i-th noisy sample images at level i resolution contained in the sample image set. It is understood that different level 0 noisy sample images often have different image features and probability distributions. Therefore, each level of noisy sample image in the sample image set contains multiple image features (because all noisy sample images at each level are generated from multiple level 0 noisy sample images through noise addition and downsampling). Therefore, this method introduces a random factor when selecting at least one level i-th noisy sample image for training the diffusion model, making the selected samples somewhat random. This helps to eliminate the particularity of a single sample and ultimately improves the generalization performance of the diffusion model, meaning that the model can be used to generate various types of noise-free images.
[0041] In one embodiment, the ability of the model to generate high-resolution images can be progressively optimized through upsampling during the backdiffusion process. For example, the i-th level noisy sample image to be input into the diffusion model can be upsampled to obtain an (i-1)-th level intermediate sample image. This image is then input into the model to obtain the predicted sample noise at the (i-1)-th level resolution output by the model. The model parameters are then adjusted based on the deviation between this predicted sample noise and the (i-1)-th level true sample noise at the (i-1)-th level resolution. Alternatively, the i-th level noisy sample image can be input into the diffusion model to obtain the i-th level predicted sample noise output by the model. The i-th level predicted sample noise is then upsampled to obtain the (i-1)-th level predicted sample noise. Finally, the parameters of the diffusion model are adjusted according to the deviation between the (i-1)-th level true sample noise and the (i-1)-th level predicted sample noise. The upsampling factor is equal to the multiple of the (i-1)-th level resolution and the i-th level resolution to ensure that the image and noise sizes match before and after sampling.
[0042] Upsampling operations require the use of upsampling operators to improve image resolution. In one embodiment, common upsampling operators can be used, such as transposed convolution (or deconvolution, which achieves upsampling by transposing the convolution kernel and performing convolution on the input feature map, inserting zero values at appropriate positions in the input feature map based on parameters like kernel size and stride, thereby increasing the feature map size), pixel shuffle (grouping the input feature map by channel and reshaping it, then transforming the channel information into spatial information through dimensionality transformation and element rearrangement, achieving spatial upsampling of the feature map), upsampling pooling (where the downsampling operator is a pooling operator, pre-recording the position information of the maximum or average value within each pooling region during pooling, and then placing the corresponding values back to their original positions during upsampling, while filling other positions with zero values or performing appropriate interpolation), and nearest neighbor interpolation. Interpolation (which directly assigns the value of each pixel in the original image to its nearest neighbor pixel in the magnified image, i.e., finding the nearest neighbor pixel in the original image and assigning its gray value to the target pixel) and bilinear interpolation (which, for each pixel in the target image, finds its four nearest neighbors in the original image and then performs linear interpolation based on the gray values of these four pixels to obtain the target pixel's value) are also possible operators. Preferably, the aforementioned upsampling operators can also be linear or differentiable operators to facilitate gradient optimization of the diffusion model during backdiffusion.
[0043] In one embodiment, when inputting a noisy sample image of level i at time step i and level i resolution into the diffusion model to obtain the noise of the predicted sample i output by the model, the resolution of the corresponding image can be specified to the diffusion model through time step embedding or resolution embedding. For example, a first vector can be constructed based on time step i and the noisy sample image of level i, and the first vector can be input into the diffusion model, thus specifying the resolution to the diffusion model through time step. Alternatively, to specify the resolution more directly, a second vector can be constructed based on time step i, descriptive information of level i resolution (such as resolution level or value), and the noisy sample image of level i, and the second vector can be input into the diffusion model. Further, a third vector output by the model to characterize the noise of the predicted sample i can be obtained, and the noise of the predicted sample i can be determined by parsing this vector.
[0044] In one embodiment, adaptive optimization algorithms such as Adam or AdamW can be used during backdivination to improve training stability and convergence speed. Additionally, methods such as weight decay and gradient clipping can be employed during backdivination to prevent overfitting and gradient explosion. Specific implementation details can be found in relevant technical documents and will not be elaborated here.
[0045] In one embodiment, during the backdiffusion process, let's assume the (j-1)th level denoised image is x. t-1 The upsampling operator is Up(x) t ), where the j-th level denoised image is shown in equation (2): p(x t-1 |x t )=N(x t-1 Up(x) t )+(1+β t )εθ(x t ,t),β t I) (2)
[0046] Where, β t This is a noise scheduling parameter used to control the noise intensity at each step. εθ(x) t ,t) represents the noise predicted by the diffusion model, i.e., the noise of the predicted samples output by the model.
[0047] In one embodiment, the model loss can be calculated based on the deviation between the j-th level predicted sample noise and the j-th level real sample noise, and the parameters of the diffusion model can be adjusted based on the model loss. The training objective is to minimize the difference between the predicted noise output by the diffusion model and the real noise added during the forward diffusion process, thereby optimizing the noise prediction capability of the diffusion model. To this end, the loss function can be set as follows (3): Loss=Ex0,εt[||εθ(x t ,t)-εt||2] (3)
[0048] During the backdiffusion process, the upsampling operator Up(x) t The noise εθ(x) is used to improve image resolution, while the noise predicted by the denoising model is... t ,t) is used to gradually restore a noise-free image.
[0049] In addition, during backdiffusion, single-step parameter updates can be performed as needed (i.e., calculating the loss function and updating the parameters immediately after inputting a sample into the model) to ensure timely parameter updates, but this often involves a large number of updates. Alternatively, multi-step aggregated parameter updates can be used (i.e., calculating the loss function and updating the parameters uniformly after inputting multiple samples into the model in batches). This method involves fewer updates but has a certain lag and averaging effect. The appropriate parameter update method can be flexibly selected according to actual needs and performance indicators. The embodiments in this specification do not limit this approach.
[0050] As can be seen from the above embodiments, this specification improves the training and inference methods of conventional diffusion models. The conventional diffusion model training process consists of a forward diffusion process (constructing training samples by progressively adding noise, i.e., gradually converting the original data into noise) and a backward diffusion process (training the model by progressively denoising, i.e., gradually recovering the original data from the noise). This scheme adds downsampling processing to the conventional forward diffusion process, thereby constructing noisy sample images with different resolutions and noise levels. These downsampled noisy sample images are then used as samples in the backward diffusion process, combined with upsampling, to train the diffusion model. This enables the trained diffusion model to "directly output noisy images of different resolutions." Correspondingly, in the model inference stage, by inputting a low-resolution pure noise image into the diffusion model and specifying the corresponding time step, and performing corresponding upsampling processing during the model's progressive denoising process, a high-resolution noise-free image can be finally obtained from the model output.
[0051] Understandably, this solution integrates resolution-specific processing logic into the diffusion model by introducing downsampling during forward diffusion and upsampling during backward diffusion and inference. This allows the diffusion model to generate noise-free images from low to high resolution with only one complete training cycle, eliminating the need for multi-stage, multiple training sessions. This significantly simplifies the training process, reduces resource consumption, and improves training efficiency. Furthermore, downsampling reduces the computational load during training and inference (significantly lower at low resolution), accelerating training and inference. In the denoising process, progressive upsampling from time step T2 to T1 allows the model to better learn and recover high-resolution details, improving the quality of generated images and videos. Finally, the diffusion model itself generates high-resolution noise-free images in a single step, avoiding resolution fluctuations introduced by post-processing modules.
[0052] This concludes the introduction to the training process of the diffusion model. The diffusion model trained using the above process can be used to quickly and accurately generate high-resolution, noise-free images. The following examples illustrate this process.
[0053] Figure 4 is a flowchart of an image generation method based on a diffusion model provided in an exemplary embodiment. This method can be applied to an image generation system. As shown in Figure 4, the method includes the following step 402, wherein this step further includes step 402.
[0054] Step 402: In response to the received image generation instruction, the diffusion model is invoked to perform T2 rounds of denoising on the T2 level denoised image of pure noise in the order of time steps from T2 to 1, so as to obtain a noise-free image. The diffusion model is trained by the method described in any of the preceding embodiments.
[0055] As mentioned earlier, during the training phase, noise is added and downsampled in the order of time steps 1 to T1 to obtain training samples. Therefore, the diffusion model after training can generate noise-free images at a maximum resolution of T1. Consequently, the maximum time step T2 during the inference phase should not exceed T1. T2 can be specified by the initiator of the image generation instruction, allowing the scheme to generate noise-free images at the user's desired resolution. The initiator can be a regular user of the system or other services associated with the image generation system; this specification does not limit this.
[0056] For example, the time step T2 can be determined based on the mapping relationship between time steps and resolution levels, according to the target resolution specified in the image generation instruction. This mapping relationship between time steps and resolution levels can be pre-created and saved during the training phase of the diffusion model (specifically, the aforementioned forward diffusion process), as exemplified in Table 1 above. In this way, the initiator can directly instruct the diffusion model to generate a noise-free image of the corresponding resolution via the image generation instruction.
[0057] The T2-round denoising process is performed on the T2-level denoised image of pure noise in the order of time step from T2 to 1. That is, the T2-level denoised image of pure noise is denoised T2 times in sequence, and the noise signal in the image is removed step by step, so as to finally obtain a clean, noise-free image with T2-level resolution.
[0058] Step 4022, wherein for an integer j satisfying 1≤j≤T2, the j-th level denoised image is denoised, including: inputting the j-th level denoised image at time step j and the j-th level resolution into the diffusion model to obtain the j-th level predicted noise output by the model, and determining the (j-1)-th level denoised image at the (j-1)-th level resolution based on the j-th level denoised image and the j-th level predicted noise by upsampling, wherein the 0-th level denoised image determined when j=1 is used as the noise-free image.
[0059] It is understood that inputting the time step j into the diffusion model is used to specify the maximum time step T2 to control the model to perform T2 rounds of denoising processing.
[0060] In one embodiment, when determining the (j-1)th level denoised image with a resolution of (j-1)th level by upsampling based on the j-th level denoised image and the j-th level predicted noise, the upsampling factor can be determined first according to the j-th level resolution and the (j-1)th level resolution. For example, the multiple (including the row multiple and column multiple) between the (j-1)th level resolution and the j-th level resolution can be determined as the upsampling factor.
[0061] Then, the j-th level denoised image and the j-th level prediction noise can be upsampled according to the upsampling factor to obtain an intermediate denoised image and intermediate prediction noise respectively (i.e., the j-th level denoised image is upsampled to obtain an intermediate denoised image, and the j-th level prediction noise is upsampled to obtain intermediate prediction noise). The intermediate prediction noise is then removed from the intermediate denoised image to obtain the (j-1)-th level denoised image at the (j-1)-th level resolution. Alternatively, the j-th level prediction noise can be removed from the j-th level denoised image at the j-th level resolution to obtain an intermediate denoised image, and then the intermediate denoised image can be upsampled according to the upsampling factor to obtain the (j-1)-th level denoised image at the (j-1)-th level resolution.
[0062] In one embodiment, if the image generation instruction does not specify specific content to be generated, the diffusion model can automatically generate a noise-free image based on the noise Gaussian distribution learned during the training phase; however, the content of this image is unpredictable. Alternatively, the initiator of the image generation instruction can specify the specific content of the noise-free image to be generated. For example, image guidance information can be determined based on the image generation instruction, and during the T2 round of denoising, the image guidance information can be used to determine the denoising direction of the corresponding image, thereby guiding the diffusion model to gradually generate a noise-free image containing specific content in a specific direction. For instance, if the user-initiated image generation instruction contains the text "dog wearing sunglasses," this text or the semantic information extracted from it can be used as image guidance information to instruct the diffusion model to ultimately generate a noise-free image containing the content "dog wearing sunglasses."
[0063] The method of determining the denoising direction of the corresponding image using the image guidance information can be achieved through conventional guidance (Vanilla Guidance, which uses image guidance information / conditional information as part of the model input, allowing the model to learn to generate the corresponding image based on the conditions during training), classifier guidance (pre-training a classifier and using the classifier's gradient to adjust the generation direction during the reverse process), and classifier-free guidance (not requiring a separate classifier, implicitly guiding the generation direction through conditional and unconditional generation during training). For details, please refer to the relevant technical records, which will not be elaborated here.
[0064] In one embodiment, the noise-free image generated can be an independent image or a video frame image (in which case multiple video frames are arranged in sequence to form a video, i.e., the model can be used to generate a video).
[0065] After generating a noise-free image in the above manner, the image can be provided to the initiator of the image generation instruction for its use, such as display or playback.
[0066] Figure 5 is a schematic structural diagram of a device provided in an exemplary embodiment. Referring to Figure 5, at the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, memory 508, and non-volatile memory 510, and may also include other hardware required for its functions. One or more embodiments of this specification can be implemented in software, for example, the processor 502 reads the corresponding computer program from the non-volatile memory 510 into memory 508 and then runs it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0067] Please refer to Figure 6. The training device for the diffusion model can be applied to the device shown in Figure 5 to implement the technical solution of this specification. The device may include: a forward diffusion unit 601, used to add noise from levels 1 to T1 of the real sample noise to the noiseless level 0 noisy sample image in the order of time steps from 1 to T1 during the forward diffusion process, and then downsample the images to obtain noisy sample images from levels 1 to T1 with progressively decreasing resolution, where T1 is a positive integer greater than 1; and a reverse diffusion unit 602, used to train the diffusion model using the noisy sample images from levels 1 to T1 during the reverse diffusion process, until the stopping condition is met. Specifically, for an integer i satisfying 1 ≤ i ≤ T1, time step i and the level i noisy sample image at level i resolution are input into the diffusion model to obtain the level i predicted sample noise output by the model. The deviation between the level i-1 real sample noise and the level i-1 predicted sample noise obtained by upsampling the level i predicted sample noise is used to adjust the parameters of the diffusion model.
[0068] Optionally, the forward diffusion unit 601 is specifically used to: add level 1 to level 1 real sample noise to the noiseless level 0 noise sample image in the order of time step 1 to T1, and downsample the noise addition result of each level to obtain the noise-added sample image of the corresponding level; or, downsample the noiseless level 0 noise sample image in the order of time step 1 to T1, and add the real sample noise of the corresponding level to the downsampled result of each level to obtain the noise-added sample image of the corresponding level.
[0069] Optionally, the reverse diffusion unit 602 is specifically used to: construct a first vector based on time step i and the i-th level noisy sample image, and input the first vector into the diffusion model; or, construct a second vector based on time step i, the description information of the i-th level resolution and the i-th level noisy sample image, and input the second vector into the diffusion model; and obtain a third vector output by the model to characterize the noise of the i-th level prediction sample.
[0070] Optionally, the number of level 0 noise-added sample images is multiple, and the level 1 to T1 noise-added sample images obtained by adding noise to each level 0 noise-added sample image and downsampling constitute a sample image set. The reverse diffusion unit 602 is specifically used to: randomly select at least one level 1 noise-added sample image from all level 1 noise-added sample images of level 1 resolution contained in the sample image set.
[0071] Optionally, at least one of the downsampling operator used for downsampling and the upsampling operator used for upsampling is a linear operator or a differentiable operator.
[0072] Please refer to Figure 7. The image generation device for the oil diffusion model can be applied to the device shown in Figure 5 to implement the technical solution of this specification. The device may include: a model invocation unit 701, used to, in response to a received image generation command, invoke the diffusion model to perform T2 rounds of denoising processing on the T2-level denoised image of pure noise in the order of time steps from T2 to 1, to obtain a noise-free image. The diffusion model is trained using the method described in any of the preceding embodiments. Specifically, for an integer j satisfying 1 ≤ j ≤ T2, the model invocation unit 701 is used to: input the time step j and the j-th level resolution of the j-th level denoised image into the diffusion model to obtain the j-th level predicted noise output by the model; and, based on the j-th level denoised image and the j-th level predicted noise, determine the (j-1)-th level denoised image of the (j-1)-th level resolution through upsampling, where the 0-th level denoised image determined when j = 1 is used as the noise-free image.
[0073] Optionally, the model invocation unit 701 is specifically used to: determine the time step T2 based on the mapping relationship between the time step and the resolution level according to the target resolution specified by the image generation instruction.
[0074] Optionally, the model invocation unit 701 is specifically configured to: determine an upsampling factor based on the j-th level resolution and the (j-1)-th level resolution; perform upsampling processing on the j-th level denoised image and the j-th level prediction noise according to the upsampling factor to obtain an intermediate denoised image and intermediate prediction noise respectively, and remove the intermediate prediction noise from the intermediate denoised image to obtain the (j-1)-th level denoised image at the (j-1)-th level resolution; or, remove the j-th level prediction noise from the j-th level denoised image at the j-th level resolution to obtain an intermediate denoised image, and perform upsampling processing on the intermediate denoised image according to the upsampling factor to obtain the (j-1)-th level denoised image at the (j-1)-th level resolution.
[0075] Optionally, a direction guidance unit 702 is also included, which is used to: determine image guidance information according to the image generation instruction, and determine the denoising direction of the corresponding image using the image guidance information when performing the T2 round denoising processing.
[0076] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.
[0077] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.
[0078] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.
Claims
1. A training method for a diffusion model, comprising a forward diffusion process and a backward diffusion process, wherein: During the forward diffusion process, noise from the first to the first level (T1) real samples is added to the noiseless first level 0 sample image in the order of time steps from 1 to T1 and then downsampled to obtain the first to the first level (T1) noisy sample images with progressively lower resolution. T1 is a positive integer greater than 1. During the back diffusion process, the diffusion model is trained using the noisy sample images of levels 1 to T1 until the stopping condition is met. For an integer i satisfying 1 ≤ i ≤ T1, the noisy sample image of level i at time step i and level i resolution is input into the diffusion model to obtain the predicted sample noise of level i output by the model. The deviation between the real sample noise of level i-1 and the predicted sample noise of level i-1 obtained by upsampling the predicted sample noise of level i is used to adjust the parameters of the diffusion model.
2. The method according to claim 1, wherein the step of adding noise from the first to T1 levels of real sample noise to the noiseless first-level 0th-level noisy sample image in order from time step 1 to T1 and then downsampling to obtain the first to T1 levels of noisy sample images with progressively decreasing resolution includes: Following the time steps from 1 to T1, noise from real samples at levels 1 to T1 is added sequentially to the noiseless level 0 sample image. The noise from each level is then downsampled to obtain the corresponding level of noise-added sample image; or... The noiseless level 0 sample image is downsampled step by step in the order of time step 1 to T1, and the corresponding level of real sample noise is added to each downsampled result to obtain the corresponding level of noise-added sample image.
3. The method according to claim 1, wherein inputting the noisy sample image of time step i and resolution i-th level into the diffusion model to obtain the predicted sample noise of the i-th level output by the model comprises: A first vector is constructed based on time step i and the i-th level noisy sample image, and the first vector is input into the diffusion model; Alternatively, a second vector can be constructed based on time step i, the description information of the i-th level resolution, and the i-th level noisy sample image, and the second vector can be input into the diffusion model; Obtain the third vector output by the model to characterize the noise of the i-th level prediction sample.
4. The method according to claim 1, wherein the number of level 0 noisy sample images is multiple, and level 1 to T1 noisy sample images are obtained by adding noise to each level 0 noisy sample image and downsampling them respectively, constituting a sample image set; and determining the level i noisy sample image with the level i resolution during the back diffusion process includes: From all the i-th level noisy sample images of the i-th level resolution contained in the sample image set, at least one i-th level noisy sample image is randomly selected.
5. The method according to claim 1, wherein at least one of the downsampling operator used for downsampling and the upsampling operator used for upsampling is a linear operator or a differentiable operator.
6. An image generation method based on a diffusion model, the method comprising: In response to the received image generation instruction, the diffusion model is invoked to perform T2 rounds of denoising on the T2 level denoised image of pure noise in the order of time steps from T2 to 1, so as to obtain a noise-free image. The diffusion model is trained by the method of any one of claims 1 to 5. Specifically, for integer j satisfying 1 ≤ j ≤ T2, the j-th level denoised image is denoised, including: The denoised image at time step j and resolution j is input into the diffusion model to obtain the predicted noise at level j output by the model. Based on the denoised image at level j and the predicted noise at level j, the denoised image at resolution j-1 is determined by upsampling. The denoised image at level j-1 determined when j=1 is used as the noise-free image.
7. The method according to claim 6, determining the largest time step T2 in the denoising process, comprising: The time step T2 is determined based on the target resolution specified by the image generation instruction and the mapping relationship between the time step and the resolution level.
8. The method according to claim 6, wherein determining the (j-1)th level denoised image with a resolution of (j-1)th level by upsampling based on the j-th level denoised image and the j-th level predicted noise comprises: The upsampling factor is determined based on the j-th level resolution and the (j-1)-th level resolution. The j-th level denoised image and the j-th level predicted noise are upsampled according to the upsampling factor to obtain an intermediate denoised image and intermediate predicted noise, respectively. The intermediate predicted noise is then removed from the intermediate denoised image to obtain the (j-1)-th level denoised image with the (j-1)-th level resolution; or... The j-th level predicted noise is removed from the j-th level denoised image at the j-th level resolution to obtain an intermediate denoised image, and the intermediate denoised image is upsampled according to the upsampling factor to obtain the (j-1)-th level denoised image at the (j-1)-th level resolution.
9. The method according to claim 6, further comprising: Image guidance information is determined according to the image generation instruction, and the denoising direction of the corresponding image is determined using the image guidance information during the T2 round of denoising processing.
10. An electronic device, characterized in that, include: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method as described in any one of claims 1-9 by executing the executable instructions.
11. A computer-readable storage medium, characterized in that, It stores computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-9.
12. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-9.
