Image generation method, image generation model training method, and apparatus

By combining text and image information to update the noise mode using a three-branch image generation model and fine-tuning the model using a pre-trained model, the problem of insufficient generalization ability of existing image super-resolution generation models in complex scenes is solved, and high-quality image generation and generalization ability are improved.

WO2026237703A1PCT designated stage Publication Date: 2026-11-19HUAWEI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2026/093163
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-15
Filing Date
2026-04-27
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Existing image super-resolution generation models lack generalization ability when faced with complex scenes and images with various types of degradation, making it difficult to generate high-quality super-resolution images.

Method used

A three-branch image generation model is adopted, including a text branch, an image branch, and a noise branch. The noise modality data is updated by combining text and image information through attention operations. The pre-trained model is used and fine-tuned to dynamically adjust the mask ratio and optimize the transition of the model from text-to-image and image-to-image tasks.

Benefits of technology

It improves the quality of image generation and the model's generation capability, enhances the generalization ability to low-resolution degradation, and improves training efficiency and generation effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2026093163_19112026_PF_FP_ABST
    Figure CN2026093163_19112026_PF_FP_ABST
Patent Text Reader

Abstract

The present application provides an image generation method, an image generation model training method, and an apparatus. The image generation method comprises: on the basis of first text and a first image, performing iterative denoising processing on an initial noisy representation in a plurality of time steps, so as to generate a second image, wherein in the process of performing the denoising processing on a noisy representation of a time step t among the plurality of time steps, an attention operation is performed on a first text representation of the first text and a first noisy representation, to obtain a first operation result; and an attention operation is performed on a first image representation of the first image and a second noisy representation, so as to obtain a second operation result. A fusion result of the two operation results is used for determining a denoising result of the time step t, thereby determining the second image. The first noisy representation and the second noisy representation are obtained by processing the noisy representation of the time step t, t being a positive integer. The solution of the embodiments of the present application facilitates improvement of the quality of a generated image.
Need to check novelty before this filing date? Find Prior Art

Description

Methods for generating images, training methods and devices for image generation models

[0001] This application claims priority to Chinese Patent Application No. 202510630363.1, filed on May 15, 2025, with the China National Intellectual Property Administration, entitled “Method for Generating Images, Training Method and Apparatus for Image Generation Model”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence, and more specifically, to a method for generating images, a training method for an image generation model, and an apparatus. Background Technology

[0003] Visual processing tasks are central to computer vision, aiming to analyze, enhance, or understand images or videos using algorithms. In the field of image generation, traditional generative models such as diffusion models have achieved some success in image and video generation, but still have limitations. Taking image super-resolution (SR) as an example, the goal of image super-resolution is to recover high-resolution (HR) images from low-resolution (LR) images, improving detail clarity. To achieve super-resolution, various technical solutions have been proposed. Due to the excellent performance of diffusion models in image generation tasks, diffusion-based methods have been introduced into the field of super-resolution. These methods typically follow the ControlNet paradigm. This is a controllable diffusion model, which introduces an independent branch alongside the basic diffusion model, utilizing processed low-resolution latent representations and related semantic cues to customize the target high-resolution image. However, current solutions struggle to guarantee the quality of image recovery. Especially in blind super-resolution tasks, current solutions lack generalization ability when facing complex scenes and images with diverse degradation types, making it difficult to guarantee the generation of high-quality super-resolution images under various conditions.

[0004] Therefore, improving the quality of generated images has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides a method for generating images, a training method for an image generation model, and an apparatus, which helps to improve the quality of the generated images.

[0006] In a first aspect, an image generation method is provided, comprising: acquiring a first image; acquiring first text, the first text indicating the content in a second image to be generated; iteratively denoising an initial noisy representation based on the first text and the first image at multiple time steps to generate a second image, the resolution of the second image being greater than the resolution of the first image, wherein, during the denoising process of the noisy representation at time step t in the multiple time steps, a first attention operation is performed on the first text representation and the first noisy representation of the first text through a first attention layer in an image generation model to obtain a first operation result; a second attention operation is performed on the first image representation and the second noisy representation of the first image through a second attention layer in an image generation model to obtain a second operation result, the fusion result of the first operation result and the second operation result is used to determine the denoising result at time step t, the denoising result at time step t is used to determine the second image, the first noisy representation and the second noisy representation are obtained by processing the noisy representation at time step t, the noisy representation at time step t being the initial noisy representation, or the noisy representation at time step t being obtained by denoising the initial noisy representation, where t is a positive integer.

[0007] According to the scheme of this application embodiment, attention operation is used to combine text information and image information to update the data of the noise mode. This allows for better reference to the information in the text space and image space to update the data of the noise mode, thereby improving the update effect of the noise mode representation and thus improving the image generation effect.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the image generation model further includes a first text branch, a first image branch, and a first noise branch. The first text representation of the first text is obtained by processing the initial text representation of the first text through the first text branch. The first image representation of the first image is obtained by processing the initial image representation of the first image through the first image branch. The first noisy representation is obtained by processing the noisy representation at time step t through the first sub-branch of the first noise branch. The second noisy representation is obtained by processing the noisy representation at time step t through the second sub-branch of the first noise branch.

[0009] In this embodiment, each branch processes the data of its corresponding modality, so that the data of each modality can work in its own space. At the same time, the text branch and the image branch serve as two conditional branches of the noise branch. In this way, when the noise branch updates the representation of the noise modality, it can refer to the processing results of the text branch and the image branch, which is conducive to better balancing the injection of text information and image information, thereby improving the updating effect of the noise modality representation and thus improving the image generation effect.

[0010] Meanwhile, in the solution of this application embodiment, a three-branch dual-condition injection structure is adopted, with the text branch and the image branch sharing the noise branch, resulting in a simple structure.

[0011] Meanwhile, the three branches can process the representation of the corresponding modes in parallel, which is beneficial to improving the model's generation ability and convergence speed.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the first text branch, the first image branch, and the first noise branch are constructed based on the transformation transformer layer.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, the image generation model includes a first diffusion block and a second diffusion block. The first diffusion block and the second diffusion block have the same structure. The first diffusion block includes a first text branch, a first noise branch, a first image branch, a first attention layer, and a second attention layer. During the denoising process of the noisy representation at time step t, the input of the second diffusion block includes a second text representation of the first text, a second image representation of the first image, and a third noisy representation. The second text representation of the first text is obtained by processing the result of the first operation through the first text branch; the second image representation of the first image is obtained by processing the result of the second operation through the first image branch; and the third noisy representation is obtained by processing the fusion result of the first operation result and the second operation result through the first noise branch.

[0014] In the scheme of this application embodiment, the representation of each modality is updated by each branch respectively, so that the representation of each modality can work in its own space. At the same time, attention operation can also combine information from other spaces, which is beneficial to better update the representation of each modality.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, the first text is the descriptive text of the first image.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, the image generation model is obtained by adjusting the parameters of the initial image generation model based on a training dataset with the goal of reducing diffusion loss. The training dataset includes multiple image-text pairs, each consisting of a matching sample image, a target sample image, and sample text. The resolution of the sample image is smaller than that of the target sample image. The diffusion loss includes the difference between the predicted noise at time step t' and the actual noise at time step t', and / or the difference between the denoising result at time step t' and the actual noisy representation at time step t'+1. Time step t' belongs to multiple time steps. The true noisy representation of time step t' is obtained by adding noise to the image representation of the target sample image. t' is a positive integer, and t'+1 belongs to multiple time steps. The true noisy representation of time step t'+1 is obtained by adding noise to the image representation of the target sample image, or the true noisy representation of time step t'+1 is the image representation of the target sample image. The predicted noise of time step t' and / or the denoising result of time step t' are obtained by processing the true noisy representation of time step t' by the initial image generation model based on the initial image representation of the sample image and the initial text representation of the sample text.

[0017] In conjunction with the first aspect, in some implementations of the first aspect, the initial image generation model is built based on a pre-trained model. The image generation model is obtained by adjusting the parameters of the initial image generation model other than the parameters of the pre-trained model with the goal of reducing diffusion loss. The pre-trained model is used to perform the text-to-image task.

[0018] In the scheme of this application embodiment, an initial image generation model is constructed based on a pre-trained model, and the parameters of the pre-trained model are frozen. This can make full use of the generation prior of the pre-trained model, such as the generation prior of the text-to-image model (e.g., text information), which is conducive to further improving the performance of the image SR task and improving the quality of the generated image. At the same time, fine-tuning is performed on the pre-trained model, which is conducive to improving training efficiency.

[0019] In conjunction with the first aspect, in some implementations of the first aspect, the first text branch, the first sub-branch, and the first attention layer belong to the pre-trained model, while the first image branch, the second sub-branch, and the second attention layer do not belong to the pre-trained model.

[0020] In conjunction with the first aspect, in some implementations of the first aspect, in the p-th iteration of the training process, the initial image representation of the sample image is generated based on the masking result, which is obtained by performing a masking operation on the sample image. The masking ratio used in the masking operation is negatively correlated with the value of p, where p is a positive integer.

[0021] In the scheme of this application embodiment, the mask ratio is dynamically adjusted so that the model can smoothly migrate from text-based graph model to graph-based graph model, or in other words, optimize the transition of the model from text-based graph task to graph-based graph task, which is beneficial to improve training speed and training effect, while enhancing the model's generalization ability to low-resolution degradation.

[0022] Secondly, a training method for an image generation model is provided. This method includes: acquiring a training dataset comprising multiple image-text pairs, each pair including a matching sample image, a target sample image, and sample text, wherein the resolution of the sample images is lower than the resolution of the target sample images; training an initial image generation model based on the training dataset to obtain a trained image generation model, wherein the image generation model iteratively denoises the initial noisy representation at multiple time steps based on the input image and the input text to the image generation model to obtain a generated image. The initial image generation model includes a first attention layer and a second attention layer. During training, the first attention layer is used to perform first attention layer on the first noisy representation of the sample text. The text representation and the fourth noisy representation undergo a first attention operation to obtain a third operation result; a second attention operation is performed on the first image representation and the fifth noisy representation of the sample image through a second attention layer to obtain a fourth operation result; the predicted noise of time step t' in multiple time steps is determined based on the fusion result of the third operation result and the fourth operation result, and / or the denoising result of time step t'. The predicted noise of time step t' and / or the denoising result of time step t' are used to adjust the parameters of the initial image generation model. The fourth noisy representation and the fifth noisy representation are obtained by processing the true noisy representation of time step t', where t' is a positive integer. The true noisy representation of time step t' is obtained by adding noise to the image representation of the target sample image.

[0023] According to the scheme of this application embodiment, attention operation is used to combine text information and image information to update the data of the noise mode. This allows for better reference to the information in the text space and image space to update the data of the noise mode, thereby improving the update effect of the noise mode representation and thus improving the image generation effect.

[0024] In conjunction with the second aspect, in some implementations of the second aspect, the initial image generation model is trained based on the training dataset, including: adjusting the parameters of the initial image generation model with the goal of reducing diffusion loss, wherein the diffusion loss includes the difference between the predicted noise at time step t' and the true noise at time step t', and / or the difference between the denoising result at time step t' and the true noisy representation at time step t'+1, wherein the true noisy representation at time step t'+1 is obtained by adding noise to the image representation of the target sample image, or the true noisy representation at time step t'+1 is the image representation of the target sample image.

[0025] In conjunction with the second aspect, in some implementations of the second aspect, the initial image generation model includes a first text branch, a first image branch, and a first noise branch. The first text representation of the sample text is obtained by processing the initial text representation of the sample text through the first text branch. The first image representation of the sample image is obtained by processing the initial image representation of the sample image through the first image branch. The fourth noisy representation is obtained by processing the true noisy representation at time step t' through the first sub-branch of the first noise branch. The fifth noisy representation is obtained by processing the true noisy representation at time step t' through the second sub-branch of the first noise branch.

[0026] In conjunction with the second aspect, in some implementations of the second aspect, the initial image generation model is built based on a pre-trained model used to perform the text-to-image task, and the parameters of the initial image generation model are adjusted with the goal of reducing diffusion loss, including: adjusting other parameters in the initial image generation model besides the parameters of the pre-trained model with the goal of reducing diffusion loss, and freezing the parameters of the pre-trained model.

[0027] In the scheme of this application embodiment, an initial image generation model is constructed based on a pre-trained model, and the parameters of the pre-trained model are frozen. This can make full use of the generation prior of the pre-trained model, such as the generation prior of the text-to-image model (e.g., text information), which is conducive to further improving the performance of the image SR task and improving the quality of the generated image. At the same time, fine-tuning is performed on the pre-trained model, which is conducive to improving training efficiency.

[0028] In conjunction with the second aspect, in some implementations of the second aspect, the first text branch, the first sub-branch, and the first attention layer belong to the pre-trained model, while the first image branch, the second sub-branch, and the second attention layer do not belong to the pre-trained model.

[0029] In conjunction with the second aspect, in some implementations of the second aspect, the initialization of the parameters of the first image branch is based on the parameters of the first sub-branch.

[0030] This allows for full utilization of the prior knowledge of the pre-trained model, which helps improve the model's convergence speed and generation performance.

[0031] In conjunction with the second aspect, in some implementations of the second aspect, the parameters of the output layer of the second attention layer are initialized with zero initialization.

[0032] This helps ensure a seamless transition from text-to-image tasks to image-to-image tasks.

[0033] In conjunction with the second aspect, in some implementations of the second aspect, the training method further includes: performing a masking operation on the sample image in the p-th iteration of the training process to obtain a masking result, where p is a positive integer; and generating an initial image representation of the sample image based on the masking result.

[0034] In conjunction with the second aspect, in some implementations of the second aspect, the mask ratio used in the masking operation is negatively correlated with the value of p.

[0035] In the scheme of this application embodiment, the mask ratio is dynamically adjusted so that the model can smoothly migrate from text-based graph model to graph-based graph model, or in other words, optimize the transition of the model from text-based graph task to graph-based graph task, which is beneficial to improve training speed and training effect, while enhancing the model's generalization ability to low-resolution degradation.

[0036] In conjunction with the second aspect, in some implementations of the second aspect, when p is less than the first threshold, the mask ratio used in the masking operation is negatively correlated with the value of p, and / or, when p is greater than or equal to the first threshold, the mask ratio used in the masking operation is the first ratio.

[0037] In conjunction with the second aspect, in some implementations of the second aspect, the sample text is the descriptive text of the sample image, and / or the descriptive text of the target sample image.

[0038] Thirdly, an image generation model is provided. This model iteratively denoises an initial noisy representation based on a first text and a first image at multiple time steps to generate a second image. The resolution of the second image is greater than that of the first image. The image generation model includes a first attention layer and a second attention layer. During the denoising process of the noisy representation at time step t, the first attention layer performs a first attention operation on the first text representation and the first noisy representation of the first text to obtain a first operation result. The second attention layer performs a second attention operation on the first image representation and the second noisy representation of the first image to obtain a second operation result. The fusion result of the first and second operation results is used to determine the denoising result at time step t. The denoising result at time step t is used to determine the second image. The first and second noisy representations are determined based on the noisy representation at time step t, which is either the initial noisy representation or the noisy representation at time step t is obtained by denoising the initial noisy representation. Here, t is a positive integer.

[0039] In conjunction with the third aspect, in some implementations of the third aspect, the image generation model further includes: a text processing module for preprocessing the first text to obtain an initial text representation of the first text, the first text being used to indicate the content in the second image to be generated; and an image processing module for preprocessing the first image to obtain an initial image representation of the first image.

[0040] In conjunction with the third aspect, in some implementations of the third aspect, the image generation model further includes a first text branch, a first image branch, and a first noise branch. The first text representation of the first text is obtained by processing the initial text representation of the first text through the first text branch. The first image representation of the first image is obtained by processing the initial image representation of the first image through the first image branch. The first noisy representation is obtained by processing the noisy representation at time step t through the first sub-branch of the first noise branch. The second noisy representation is obtained by processing the noisy representation at time step t through the second sub-branch of the first noise branch.

[0041] In conjunction with the third aspect, in some implementations of the third aspect, the image generation model includes a first diffusion block and a second diffusion block. The first diffusion block includes a first text branch, a first noise branch, a first image branch, a first attention layer, and a second attention layer. The first diffusion block and the second diffusion block have the same structure. During the denoising process of the noisy representation at time step t, the input of the second diffusion block includes the second text representation of the first text, the second image representation of the first image, and the third noisy representation. The second text representation of the first text is obtained by processing the result of the first operation through the first text branch; the second image representation of the first image is obtained by processing the result of the second operation through the first image branch; and the third noisy representation is obtained by processing the fusion result of the first operation result and the second operation result through the first noise branch.

[0042] In conjunction with the third aspect, in some implementations of the third aspect, the first text is the descriptive text of the first image.

[0043] Fourthly, an apparatus for generating an image is provided, the apparatus comprising modules / units for performing the methods of the first aspect and any implementation thereof.

[0044] Fifthly, a training apparatus for an image generation model is provided, the apparatus comprising modules / units for performing the methods of the second aspect and any implementation thereof.

[0045] It should be understood that the extensions, limitations, interpretations and descriptions of the relevant content in the first aspect above also apply to the same content in the second, third, fourth and fifth aspects.

[0046] Sixthly, a computing device is provided, including a processor and a memory, and optionally, an input / output interface. The processor controls the input / output interface to send and receive information, the memory stores a computer program, and the processor retrieves and runs the computer program from the memory, enabling the execution of the method described in the first aspect, the second aspect, or any possible implementation thereof.

[0047] Optionally, the processor can be a general-purpose processor, which can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, integrated circuit, etc.; when implemented in software, the processor can be a general-purpose processor that reads software code stored in memory. This memory can be integrated into the processor or located outside the processor and exist independently.

[0048] Optionally, the aforementioned computing device may be a terminal device / server, or a chip within a terminal device / server.

[0049] In a seventh aspect, a chip is provided that acquires and executes instructions to implement the methods in the first aspect or the second aspect and any of the implementation methods described above.

[0050] Optionally, as one implementation, the chip includes a processor and a data interface, through which the processor reads instructions stored in the memory and executes the methods in the first aspect or the second aspect and any of the implementations described above.

[0051] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to perform the method in the first aspect or the second aspect and any of the implementations.

[0052] Eighthly, a computer program product containing instructions is provided, which, when executed by a computing device, cause the computing device to perform the methods described in the first or second aspect and any of their implementations.

[0053] A ninth aspect provides a computer-readable storage medium including computer program instructions that, when executed by a computing device, perform the method as described in the first or second aspect and any of its implementations.

[0054] As examples, these computer-readable storage devices include, but are not limited to, one or more of the following: read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), flash memory, electrically EPROM (EEPROM), and hard drive.

[0055] Alternatively, as one implementation method, the aforementioned storage medium can specifically be a non-volatile storage medium. Attached Figure Description

[0056] Figure 1 is a schematic block diagram of a system applicable to an embodiment of this application.

[0057] Figure 2 is a schematic diagram of a system architecture according to an embodiment of this application.

[0058] Figure 3 is a schematic diagram of a model architecture according to an embodiment of this application.

[0059] Figure 4 is a schematic structural diagram of the diffusion module according to an embodiment of this application.

[0060] Figure 5 is a schematic structural diagram of the diffusion block according to an embodiment of this application.

[0061] Figure 6 is a schematic flowchart of a training method for an image generation model according to an embodiment of this application.

[0062] Figure 7 is a schematic diagram of a training process according to an embodiment of this application.

[0063] Figure 8 is a schematic diagram of a partial structure of a pre-trained model according to an embodiment of this application.

[0064] Figure 9 is a schematic flowchart of an image generation method according to an embodiment of this application.

[0065] Figure 10 is a schematic block diagram of an apparatus according to an embodiment of this application.

[0066] Figure 11 is a schematic diagram of the architecture of a computing device according to an embodiment of this application. Detailed Implementation

[0067] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0068] The terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” and “the” are intended to include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, “at least one,” “at least one,” and “one or more” refer to one, two, or more than two. “First,” “second,” and various numerical designations are merely distinctions for descriptive convenience and are not intended to limit the scope of the embodiments of this application. “And / or” is used to describe the correspondence between corresponding objects, indicating that three relationships can exist. For example, “A and / or B” can represent: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character “ / ” generally indicates that the preceding and following related objects are in an “or” relationship. The order of the process numbers below does not imply the order of execution. The execution order of each process should be determined by its function and internal logic and should not constitute any limitation on the implementation process of the embodiments of this application. For example, in the embodiments of this application, the words "301", "401", "501" etc. are merely identifiers for the convenience of description and do not limit the order of execution steps.

[0069] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. In this application, the words "exemplary" or "for example" are used to indicate that something is illustrative, exemplary, or descriptive. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized. In the embodiments of this application, descriptions such as "when," "in the case of," "if," and "if" all refer to the fact that the device will perform a corresponding processing under certain objective circumstances, and are not a limitation on time, nor do they require the device to perform a judgment action during implementation, nor do they imply any other limitations.

[0070] In this application, "for indicating" can include both direct and indirect indication. When describing an indication message as indicating A, it can include whether the indication message directly indicates A or indirectly indicates A, but does not necessarily mean that the indication message carries A.

[0071] The solutions in this application can be applied to application scenarios involving SR reconstruction, that is, the solutions in this application can be used to perform SR reconstruction tasks. SR reconstruction tasks can be specifically applied to, but are not limited to, the following scenarios: medical imaging, satellite imaging, video enhancement, security monitoring, and photo SR reconstruction, etc.

[0072] The following uses photo SR reconstruction as an example to illustrate the application scenarios of this application embodiment.

[0073] For example, after receiving user input (e.g., a photo to be processed), the solution of this application embodiment can be used to process the user input to generate a corresponding image (e.g., a high-resolution photo), and the generated result can be presented to the user (the presentation method may include, but is not limited to, displaying, playing, saving, or uploading to the cloud).

[0074] For example, the solution of this application embodiment can be applied to an image generation application. This application can run on a terminal device or a cloud server. As another example, the solution of this application embodiment can be applied to a cloud service that provides image generation services. This cloud service can be provided by a cloud server.

[0075] Taking an application running on a terminal device as an example, a user can open the client of the application installed on the terminal device, input the photo to be processed, and the application can process the photo using the method provided in the embodiments of this application to generate a corresponding image and present the generated result to the user.

[0076] To facilitate understanding of the solutions in the embodiments of this application, the terms that may be involved in the embodiments of this application will be explained below.

[0077] (1) Neural Networks:

[0078] Neural networks can be composed of neural units, which can refer to units represented by x. s The operation unit takes an intercept of 1 as input.

[0079] A neural network is a network formed by connecting multiple individual neural units, meaning that the output of one neural unit can be the input of another. The input of each neural unit can be connected to the local receptive field of the previous layer to extract features from the local receptive field, which can be a region composed of several neural units.

[0080] (2) Deep Neural Networks:

[0081] A deep neural network (DNN), also known as a multilayer neural network, can be understood as a neural network with multiple hidden layers. Based on the position of the layers, the internal neural network of a DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. The layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer.

[0082] In deep neural networks, more hidden layers allow the network to better depict complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can accomplish more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by vectors from many layers).

[0083] (3) Loss function:

[0084] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0085] (4) Backpropagation algorithm:

[0086] Backpropagation (BP) is an algorithm used during training to correct the parameters in the initial model, thereby reducing the model's error loss. Specifically, forward propagation of the input signal to the output generates an error loss; this error loss information is then propagated back to update the parameters in the initial model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining optimal model parameters, such as the weight matrix.

[0087] (5) Diffusion model:

[0088] A diffusion model is a generative model that generates new samples by progressively adding noise to data and then training a denoising model. This model can be used for various tasks such as image generation and text-to-image generation. The diffusion model consists of two stages: a forward process (noise diffusion) and a backward process (denoising recovery). The backward process can also be called a reverse diffusion process, a sampling process, or a denoising process.

[0089] During training, the model learns how to remove noise at each time step. Given noisy data, the model predicts the noise components or directly outputs the denoised result, with the optimization objective typically being the mean squared error or another suitable loss function. After training, during the inference phase, the model can progressively remove noise from pure noise, generating new data. For example, at a given time step t in a backdiffusion process, the diffusion model can predict noise, which can be used to update the noisy data at time step t to move to the next time step t-1, thus obtaining the noisy data at time step t-1.

[0090] (6) Diffusion Transformer (DiT):

[0091] DiT is a generative model that combines the Transformer architecture with a diffusion model, primarily used for image and video generation tasks.

[0092] DiT is typically based on the latent diffusion model (LDM) framework, which means that it operates on a low-dimensional latent space compressed by a variational autoencoder (VAE), rather than directly processing pixel-level data.

[0093] Given an input image, the compressed features of the image are contaminated by the standard noise-adding process of the diffusion model to obtain noisy features. The noisy features, conditional features, and features corresponding to the ground truth are then input into DiT and the output result is completed, thus completing one DiT pre-propagation process.

[0094] DiT is composed of basic units of DiT modules. Most elements in DiT modules are similar to standard Transformer modules, including multi-head self-attention, layer normalization, feed-forward layers, and so on.

[0095] (7) ControlNet:

[0096] ControlNet is a neural network architecture that enhances the controllability of generated images by adding additional conditional control to a diffusion model. The diffusion model itself is a generative model that generates data samples, such as images, by progressively removing noise. The key to combining ControlNet with the diffusion model lies in its ability to introduce specific conditional information at each step of the diffusion process, thereby influencing the generated result.

[0097] The basic principle of ControlNet is to add a control branch to the diffusion model, enabling the model to generate images that meet specific requirements based on specific conditional information (such as edge maps, depth maps, key points, etc.). This control branch is responsible for processing the conditional information and fusing it with the generated features of the main network through residual connections at each layer of the diffusion process.

[0098] During training, ControlNet locks the parameters of the original diffusion model, updating only the weights of the control branch. This is to preserve the original capabilities of the diffusion model while allowing the control branch to learn how to adjust the generation process based on specific conditions. Initially, since the weights of the zero convolutional layers are initialized to zero, they have no impact on the generation results. As training progresses, ControlNet gradually learns how to effectively utilize conditional information to guide the generation process.

[0099] (8) Large Language Model (LLM):

[0100] Large Language Models (LLMs) are neural network models with a large number of parameters trained on vast corpora, capable of understanding and generating natural language text. Specifically, LLMs are typically based on neural network techniques, learning the syntax, semantics, and contextual information of a language through training on large amounts of text data. During training, the model continuously optimizes its parameters to improve its ability to understand and generate text. Due to their powerful ability to understand natural language, LLMs have been widely applied in many fields to solve natural language understanding and generation problems. LLMs have broad applications in artificial intelligence, such as natural language processing, machine translation, and dialogue systems.

[0101] (9) Masked image modeling (MIM) strategy:

[0102] MIM is a self-supervised method that learns complete semantic associations using partial observation data. Its core idea is to randomly mask a portion of the input image (e.g., 30%-50% of an image patch), and then train the model to predict the content of the masked region. This strategy significantly improves the model's ability to understand the semantic structure of images by explicitly modeling context dependencies.

[0103] During training, a portion of the image is randomly selected for masking (e.g., patch-based partitioning). Typically, a uniform distribution or semantically aware masking ratio is used (e.g., high-frequency regions are more easily masked).

[0104] Figure 1(a) illustrates a schematic diagram of a system applicable to the method of this application embodiment. For example, as shown in Figure 1(a), the system may include a user device and a data processing device. The user device can act as the initiator of data processing. The user can initiate a request through the user device. The user device can also be replaced by a client device. The user device can send the user's request to the data processing device for processing and display the processing result to the user. For example, the data processing device can be a device or server with data processing capabilities, such as a cloud server, network server, application server, or management server. The data processing device can also be replaced by a server device. The data processing device can receive input data from the user device through an interactive interface, such as query statements / voice / text / images, etc., and perform data processing through a storage device for storing data and a data processing processor, such as data training, machine learning, deep learning, search, reasoning, decision-making, etc., for example, executing the method of this application embodiment. The storage device can be a general term, including local storage and a database storing historical data. The database can be located on the data processing device or on other devices.

[0105] For example, user devices may include mobile phones, tablets, wearable devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), or vehicles, etc.

[0106] As an example, an image generation application can be installed on the user device, or a webpage related to image generation functionality can be displayed. These applications and webpages can provide an interface where the user device can receive data (such as images) input by the user and send the input data to a data processing device. The data processing device can execute the method described in this application's embodiments and return the result to the user device. For example, a cloud server can be used as the data processing device. The cloud service can provide image generation services, executing the scheme described in this application's embodiments.

[0107] Figure 1(b) shows a schematic diagram of a system applicable to another method of this application embodiment. For example, as shown in Figure 1(b), the data processing system may include a user equipment. The user equipment can directly function as a data processing device, receiving input data, such as query statements / voice / text / images, and performing data processing, such as executing the method of this application embodiment. The specific process is similar to that in Figure 1(a), and will not be described again here.

[0108] For example, the solutions of this application embodiment can be applied to user equipment. For instance, the model of this application embodiment can be built into the user equipment, and the user equipment can execute the methods of this application embodiment. Alternatively, the model of this application embodiment can be deployed in an application (APP) of the user equipment, that is, the APP can call and execute the methods of this application embodiment.

[0109] Figure 1(c) shows a schematic diagram of a system applicable to another embodiment of the method of this application. The user equipment in Figure 1(a) or Figure 1(b) can be the local device 201 or 202 in Figure 1(c), and the data processing device in Figure 1(a) or Figure 1(b) can be the execution device 210. The data storage system 250 can be integrated on the execution device 210 or can be located in the cloud or on other network servers.

[0110] Figure 2 illustrates a system architecture 100 provided in an embodiment of this application. In Figure 2, a data acquisition device 160 is used to acquire training data. For example, for the image generation method in this embodiment, the training data may include image pairs, wherein the image pair may include a low-resolution image and a corresponding high-resolution image.

[0111] After collecting the training data, the data acquisition device 160 stores the training data in the database 130, and the training device 120 trains the target model / rule 101 based on the training data maintained in the database 130.

[0112] The following describes how the training device 120 obtains the target model / rule 101 based on the training data. The training device 120 processes the input raw data and compares the output value with the target value until the difference between the output value of the training device 120 and the target value is less than a certain threshold, thereby completing the training of the target model / rule 101.

[0113] The target model / rule 101 described above can be used to implement data processing in the embodiments of this application, such as image generation. Specifically, the target model / rule 101 in the embodiments of this application can be a neural network model. It should be noted that in practical applications, the training data maintained in the database 130 may not all come from the data acquisition device 160; it may also be received from other devices. Furthermore, it should be noted that the training device 120 may not necessarily train the target model / rule 101 entirely based on the training data maintained in the database 130; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0114] The target model / rule 101 trained by training device 120 can be applied to different systems or devices, such as execution device 110 shown in Figure 2. Execution device 110 can be a terminal, such as a mobile phone, tablet, laptop, augmented reality (AR) / virtual reality (VR) device, in-vehicle terminal, or a server or cloud platform. In Figure 2, execution device 110 is configured with input / output (I / O) interface 112 for data interaction with external devices. Users can input data to I / O interface 112 through client device 140. In this embodiment, the input data may include an image to be processed input by the client device.

[0115] During the preprocessing of input data by the execution device 110, or during the calculation module 111 of the execution device 110 performing calculations and other related processes, the execution device 110 can call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0116] Finally, I / O interface 112 returns the processing result, such as the data processing result obtained above, to client device 140, thereby providing it to the user.

[0117] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different objectives or tasks. The corresponding target models / rules 101 can be used to achieve the above objectives or complete the above tasks, thereby providing the user with the required results.

[0118] In the scenario shown in Figure 2, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can be presented in various forms such as display, sound, or animation. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0119] It is worth noting that Figure 2 is only a schematic diagram of a system architecture provided by an embodiment of this application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 2, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 can also be placed in the execution device 110.

[0120] Current super-resolution (SR) reconstruction schemes have limitations, making it difficult to guarantee restoration quality, especially for blind super-resolution tasks. Blind super-resolution is an important branch of super-resolution tasks. Unlike traditional super-resolution methods, in blind super-resolution, the specific information about the degradation process of low-resolution images is unknown. This unknown and complex nature of degradation presents a greater challenge to high-quality image restoration. In real-world scenarios, acquired images often have low resolution due to equipment limitations, transmission losses, and environmental interference. The emergence of blind super-resolution technology offers the possibility of improving the quality of these low-resolution images, enabling them to meet subsequent analysis and application needs, and has shown broad application prospects in multiple fields such as medical imaging, satellite imaging, video enhancement, and security monitoring. However, in blind super-resolution tasks, current schemes lack generalization ability when facing complex scenes and images with diverse degradation types, making it difficult to guarantee the generation of high-quality super-resolution images under various conditions.

[0121] In view of this, embodiments of this application provide an image generation method that helps improve the quality of the generated image.

[0122] It should be understood that the images in the embodiments of this application can be single images, such as a single photograph taken by a camera, or image frames from a video. In other words, taking SR tasks as an example, the solutions in the embodiments of this application can be used for image SR tasks or video SR tasks. For ease of description, the solutions in the embodiments of this application are mainly described using images as an example, and do not constitute a limitation on the solutions in the embodiments of this application.

[0123] The solutions in this application can be applied to image generation tasks, such as image SR (Simulation / Regeneration) tasks. For ease of description, this application mainly uses image SR tasks as an example for illustration, and does not constitute a limitation on the solutions in this application. When applied to other image generation tasks, the training data and the image to be processed can be adaptively replaced.

[0124] To better illustrate the solutions of the embodiments of this application, the model architecture of the embodiments of this application will be described below.

[0125] Figure 3 illustrates a schematic diagram of a model architecture according to an embodiment of this application. The model 300 shown in Figure 3 is the image generation model in this embodiment of the application.

[0126] As shown in Figure 3, model 300 includes a text processing module 310, an image processing module 320, and a diffusion module 330.

[0127] In this case, the input to model 300 may include input text and an input image, and the output of model 300 may include an image generated by model 300. The input text is used to indicate the content of the image to be generated.

[0128] The input image is the image that needs to undergo super-resolution processing. The image to be generated is the high-resolution version of the input image that is expected to be generated. In other words, the resolution of the image to be generated is higher than the resolution of the input image.

[0129] The text processing module 310 can be used to preprocess the input text to make it conform to the input format requirements of the diffusion module 330.

[0130] The image processing module 320 can be used to preprocess the input image to make it conform to the input format requirements of the diffusion module 330.

[0131] The diffusion module 330 can be used to iteratively denoise the initial noisy representation at multiple time steps based on the output of the text processing module 310 and the output of the image processing module 320 to generate an image.

[0132] The text processing module 310 is described below.

[0133] The text processing module 310 can be used to preprocess the input text to obtain an initial text representation of the input text.

[0134] The initial text representation of the input text conforms to the input format requirements of the diffusion module 330.

[0135] For example, if the input text is text #1 (i.e., the first text), the text processing module 310 can be used to preprocess text #1 to obtain the initial text representation of text #1.

[0136] The following provides an illustrative example of the form of the initial text representation of the input text.

[0137] As one possible implementation, the text processing module 310 can be used to process the input text to obtain a text embedding of the input text. The text embedding of the input text can serve as the initial text representation of the input text.

[0138] The input to the diffusion module 330 may include embedding, in which case the text processing module 310 may be used to process the input text to obtain the text embedding of the input text.

[0139] For example, the text processing module 310 can be used to perform tokenization on the input text to obtain a sequence of words (tokens) in the input text. Each token in the token sequence is mapped to a vector space, and positional encoding is used to add positional information to each token to obtain the text embedding of the input text.

[0140] The token for the input text can be a word, phrase, or character.

[0141] In the embodiments of this application, word segmentation processing can also be replaced by other descriptions such as tokenization processing.

[0142] For ease of description, the above process of obtaining text embedding can be called the tokenization process.

[0143] The above are just examples. The text embedding of the input text can also be obtained through other methods besides tokenization. This application does not limit this.

[0144] Furthermore, the text processing module 310 can also be used to perform other processing on the input text, which is not limited in this embodiment.

[0145] In some scenarios, the input to the diffusion module 330 can be in other formats, and the text processing module 310 can be used to perform corresponding processing to match the input requirements of the diffusion module 330.

[0146] For example, the input to the diffusion module 330 can be text data, meaning the initial text representation of the input text can be the input text itself. In this case, model 300 may also exclude the text processing module 310.

[0147] The method for obtaining input text during the inference and training phases of Model 300 can be found in the following text.

[0148] The image processing module 320 is described below.

[0149] The image processing module 320 can be used to preprocess the input image to obtain an initial image representation of the input image.

[0150] The initial image representation of the input image conforms to the input format requirements of the diffusion module 330.

[0151] For example, if the input image is image #1 (i.e., the first image), the text processing module 310 can be used to preprocess image #1 to obtain the initial image representation of image #1.

[0152] The following provides an illustrative example of the form of the initial image representation of the input image.

[0153] As one possible implementation, the image processing module 320 can be used to preprocess the input image to obtain an image embedding of the input image. The image embedding of the input image can serve as an initial image representation of the input image.

[0154] The input to the diffusion module 330 may include embedding, in which case the image processing module 320 may be used to process the input image to obtain an image embedding.

[0155] For example, the image processing module 320 can be used to process the input image to obtain a token sequence of the input image. Each token in the token sequence of the input image is mapped to a vector space, and positional encoding is used to add its position information in the original image to each token to obtain an image embedding.

[0156] Specifically, the image processing module 320 can segment the input image to obtain multiple patches, flatten these patches to form a token sequence. Each patch can be considered a token. These multiple patches can be all the patches obtained from segmenting the input image, or only some of them. Then, each token in the token sequence is mapped to a vector space, and its position information in the original image is added to each token to obtain an image embedding.

[0157] Furthermore, the image processing module 320 can also encode the input image to obtain a latent representation of the input image. The latent representation of the input image is processed to obtain a token sequence of the latent representation. Each token in the token sequence is mapped to a vector space, and positional encoding is used to add its positional information in the latent representation to each token to obtain the image embedding.

[0158] The latent representation of the input image can be obtained by encoding the input image through a VAE encoder, as shown in Figure 3(c).

[0159] Latent representations can also be replaced by other descriptive methods such as latent space representations of data, latent features, latent representations, latent variables, or latent vectors.

[0160] Furthermore, the image processing module 320 can also be used to perform other processing on the input image, which is not limited in this embodiment.

[0161] In some scenarios, the input to the diffusion module 330 can be in other formats, and the image processing module 320 can perform corresponding processing to match the input requirements of the diffusion module 330.

[0162] For example, the input to the diffusion module 330 can be image data, meaning the initial text representation of the input image can be the input image itself. In this case, the model 300 may also not include the image processing module 320.

[0163] The method for obtaining input images during the inference and training phases of Model 300 can be found in the following text.

[0164] The diffusion module 330 can be used to perform iterative denoising on the initial noisy representation based on the initial text representation of the input text and the initial image representation of the input image to obtain the predicted representation, i.e. the result of the iterative denoising.

[0165] The initial noisy representation is the noisy representation of the initial time step.

[0166] In the embodiments of this application, the time step can also be replaced by a description such as time.

[0167] For example, the initial noisy representation can be initial noise. For instance, during the inference phase, the initial noise can be random noise, such as random Gaussian noise. For instance, during the training phase, the initial noise can be obtained by adding noise to the high-resolution image in the image-text pair. A more detailed description follows.

[0168] The format of the noisy representation depends on the requirements of the diffusion module 330 for the input format.

[0169] As an example, the input to the diffusion module 330 can include pixel space input, so the initial noise can be pixel space noise, meaning the dimension of the initial noise can be the same as the dimension of the image. In this case, the predicted representation can be the generated super-resolution image.

[0170] In the embodiments of this application, the image generated by model 300 can be referred to as a super-resolution image.

[0171] As another example, the input to the diffusion module 330 may include the input of a low-dimensional latent space, and the initial noise may be the noise of the latent space, the dimension of which depends on the dimension of the latent space.

[0172] Specifically, the diffusion module 330 can be based on the LDM framework. Accordingly, the diffusion module 330 operates in the low-dimensional latent space compressed by the VAE, and the diffusion process is implemented in the latent space of the VAE. In this case, the initial noisy representation is the initial noisy latent representation, and the predicted representation is the predicted latent representation. During the image generation process, the initial noisy latent representation can be iteratively denoised by the diffusion module 330 to generate the predicted latent representation, and then the predicted latent representation can be decoded by the decoder to generate the super-resolution image.

[0173] For example, a super-resolution image can be obtained by decoding the predicted latent representation through a VAE decoder, as shown in Figure 3(c).

[0174] Furthermore, the input to the diffusion module 330 can include embedding, so the initial noisy representation can also be obtained by tokenizing the initial noise. The tokenization process can be referred to the previous description, and will not be repeated here.

[0175] The iterative denoising process is explained below.

[0176] For ease of description, as shown in Figure 3(b) or (c), assume that the iterative denoising process has a total of T time steps. From time step T to time step 0, iterative denoising is continuously performed by the diffusion module 330 to obtain the denoising results of each time step (i.e., the noisy representation of each time step). The noisy representation at the initial time is the noisy representation Z of time step T. T The predicted representation can be understood as the final denoised result, i.e., the noisy representation Z0 at time step 0. T is an integer greater than 1.

[0177] During the denoising process, the denoising result at each time step can be obtained based on the output of the diffusion module 330. Specifically, for each time step, the input of the diffusion module 330 can include the initial text representation of the input text, the initial image representation of the input image, and the noisy representation of that time step. The diffusion module 330 can output the predicted noise of that time step or the denoising result of that time step. The predicted noise of a time step can be understood as the noise that needs to be removed from the noisy representation of that time step, predicted by the diffusion module 330. The denoising result of a time step can be the result after removing noise from the noisy representation of that time step, predicted by the diffusion module 330. During the inference phase, except for the initial noisy representation (i.e., the noisy representation of time step T), the noisy representation of each time step is the denoising result of the previous time step.

[0178] In a given time step t of the denoising process, the diffusion module 330 denoises the noisy representation at time step t based on the initial text representation of the input text and the initial image representation of the input image, obtaining the denoising result at time step t. Moving to the next time step t-1, the denoising result at time step t-1 can be used as the noisy representation at time step t-2. Here, t is a positive integer less than or equal to T. Repeating this process for T time steps achieves iterative denoising of the initial noisy representation.

[0179] The denoising process can also be replaced by other descriptions such as generation process, sampling process, or back diffusion process.

[0180] The structure of the diffusion module 330 is described below.

[0181] The diffusion module 330 may include one or more diffusion blocks.

[0182] For example, the diffusion module 330 may include a stacked structure of multiple diffusion blocks. In other words, the multiple diffusion blocks process the input data sequentially, meaning that the output of one diffusion block can be used as the input of another diffusion block in two adjacent diffusion blocks. Alternatively, the output of one diffusion block can be used to determine the input of another diffusion block. For example, the output of one diffusion block may be processed and then input to the next diffusion block adjacent to it.

[0183] The following explanation uses a diffusion block (such as diffusion block #1) in diffusion module 330 as an example.

[0184] The designation of diffusion block #1 is for descriptive convenience only and has no limiting effect. Diffusion block #1 can be any diffusion block in diffusion module 330.

[0185] The input to the diffusion block can include representations of multiple modalities, such as text representations of input text, image representations of input images, and noisy representations.

[0186] Taking diffusion block #1 as an example, the input of diffusion block #1 may include the text representation #1 of the input text, the image representation #1 of the input image, and the noisy representation #1.

[0187] The numbers #1 for the text representation of the input text, #1 for the image representation of the input image, and #1 for the noisy representation are for descriptive convenience only and are not intended to be limiting. For diffusion block #1, the text representation input to this diffusion block can be regarded as the text representation of the input text #1, the image representation input to this diffusion block can be regarded as the image representation of the input image #1, and the noisy representation input to this diffusion block can be regarded as the noisy representation #1.

[0188] The text representation #1 of the input text can be the initial text representation of the input text, or the text representation #1 of the input text can be determined based on the initial text representation of the input text.

[0189] The image representation #1 of the input image can be the initial image representation of the input image, or the image representation #1 of the input image can be determined based on the initial image representation of the input image.

[0190] Noisy characterization #1 can be the noisy characterization of each time step of the input diffusion module 330, or the noisy characterization #1 can be determined based on the noisy characterization of each time step of the input diffusion module 330.

[0191] Diffusion blocks can be used to process noisy representations of inputs in order to update noisy representations.

[0192] Accordingly, the output of the diffusion block includes at least the updated noisy representation, i.e., the noisy representation obtained after processing by the diffusion block. The updated noisy representation can be the denoising result and / or predicted noise at the current time step. Alternatively, the updated noisy representation can be the denoising result and / or predicted noise used to determine the current time step.

[0193] Taking diffusion block #1 as an example, the output of diffusion block #1 can include noisy representation #2 (i.e., the updated noisy representation #1).

[0194] Noisy characterization #2 can be the denoised result and / or predicted noise at the current time step. Alternatively, noisy characterization #2 can be the denoised result and / or predicted noise used to determine the current time step.

[0195] Furthermore, when the diffusion module 330 includes multiple diffusion blocks, the diffusion blocks can also be used to update the text representation of the input text and the image representation of the input image.

[0196] Accordingly, the output of the diffusion block may also include the updated text representation of the input text and the updated image representation of the input image.

[0197] Taking diffusion block #1 as an example, the output of diffusion block #1 can also include the text representation #2 of the input text (i.e., the updated text representation #1) and the image representation #2 of the input image (i.e., the updated image representation).

[0198] The following section uses a diffusion block (such as diffusion block #1) as an example to explain the input and output in detail. The input and output of other diffusion blocks can be found in the description of diffusion block #1.

[0199] For example, the input of the diffusion module 330 can be used as the input of the first diffusion block in the diffusion module 330.

[0200] Taking diffusion block #1 as an example, if diffusion block #1 is the first diffusion block of diffusion module 330, then the text representation #1 of the input text can be the initial text representation of the input text; the noisy representation #1 can be the noisy representation of the current time step; and the image representation #1 of the input image can be the initial image representation of the input image.

[0201] Alternatively, the input to the diffusion module 330 can be used to determine the input to the first diffusion block in the diffusion module 330. For example, at least one of the initial text representation of the initial text, the noisy representation of the current time step, or the initial image representation of the initial image can be processed and then input into the first diffusion block.

[0202] For example, the output of the last diffusion block in diffusion module 330 can be used as the output of diffusion module 330.

[0203] Taking diffusion block #1 as an example, if diffusion block #1 is the last diffusion block of diffusion module 330, then the noisy characterization #2 can be the predicted noise or denoising result of the current time step output by diffusion module 330.

[0204] Alternatively, the noisy representation of the output of the last diffusion block in diffusion module 330 can be used to determine the output of diffusion module 330. For example, the noisy representation of the output of the last diffusion block can be processed and used as the output of diffusion module 330.

[0205] For example, taking time step t as an example, the input to the diffusion module 330 may include the initial text representation of the input text, the initial image representation of the input image, and the noisy representation at time step t. The text representation input to the first diffusion block can be the initial text representation of the input text, the image representation input to the first diffusion block can be the initial image representation of the input text, and the noisy representation input to the first diffusion block can be the noisy representation at time step t. The noisy representation output by the last diffusion block in the diffusion module 330 can be the predicted noise at time step t or the denoised result at time step t.

[0206] As mentioned earlier, in two adjacent diffusion blocks, the output of one diffusion block can be used as the input of the other diffusion block. Alternatively, the output of one diffusion block can be used to determine the input of the other diffusion block.

[0207] If diffusion block #1 is not the first diffusion block in diffusion module 330, the input of diffusion block #1 can be the output of the diffusion block in diffusion module 330 that is adjacent to diffusion block #1 and precedes diffusion block #1, or it can be determined based on that output.

[0208] If diffusion block #1 is not the last diffusion block in diffusion module 330, the output of diffusion block #1 can be the input of the diffusion block adjacent to diffusion block #1 after diffusion block #1 in diffusion module 330, or it can be used to determine the input.

[0209] In other words, the text representation, image representation, and noisy representation output by a diffusion block can serve as three inputs to the next diffusion block adjacent to that diffusion block, or as three inputs to determine the next diffusion block.

[0210] Figure 4 shows a schematic diagram of an example structure of the diffusion module 330.

[0211] For example, taking two adjacent diffusion blocks (diffusion block 331 and diffusion block 332) in Figure 4 as an example, the data processing order in these two diffusion blocks is as follows: diffusion block 331, diffusion block 332. Diffusion block 332 is the next diffusion block adjacent to diffusion block 331.

[0212] The output of diffuser block 331 can be used as the input of diffuser block 332:

[0213] The text representation output by diffusion block 331 can be used as the text representation input to diffusion block 332;

[0214] The noisy representation output by diffuser block 331 can be used as the noisy representation input to diffuser block 332;

[0215] The image representation output by diffuser block 331 can be used as the image representation input to diffuser block 332.

[0216] Assuming that diffusion block 332 is diffusion block #1, then the text representation output by diffusion block 331 is text representation #1, the noisy representation output by diffusion block 331 is noisy representation #1, and the image representation output by diffusion block 331 is image representation #1.

[0217] The structure of the diffuser block is described below.

[0218] A diffusion block can include a text branch, a noise branch, and an image branch. If the diffusion module 330 includes multiple diffusion blocks, the multiple diffusion blocks can adopt the same structure, that is, each diffusion block can include a text branch, a noise branch, and an image branch. The parameters of the branches in different diffusion blocks can be the same or different. For example, the parameters of the text branches in two diffusion blocks can be the same or different.

[0219] The text branch can also be called the text processing branch. The noise branch can also be called the noise processing branch. The image branch can also be called the image processing branch, the low-resolution full-token processing branch, or the low-resolution branch.

[0220] If the diffusion module 330 acts on the latent space, the noise branch can also be called the noise latent variable processing branch.

[0221] These three branches are different branches.

[0222] The text branch can include multiple network layers. The image branch can include multiple network layers. The noise branch can include multiple network layers.

[0223] For any two branches, there may be a shared network layer. Two branches are different branches if at least one of the network layers in the two branches is not shared.

[0224] The text branch, noise branch, and image branch are used to process the representation of the corresponding modalities, respectively.

[0225] In this scheme, images and text can be injected as two separate streams for control, forming a three-branch structure.

[0226] The following example uses diffusion block #1 as an illustration.

[0227] Diffusion block #1 can include a text branch, a noise branch, and an image branch.

[0228] The text branch is used to process the text representation #1 of the input text to obtain the text representation #3 of the input text.

[0229] The numbers #2 and #3 for the text representation of the input text are for descriptive convenience only and have no limiting effect. For diffusion block #1, the text representation of the output diffusion block after text branching can be regarded as the text representation of the input text #2, and the text representation of the input text #3 can be regarded as the intermediate result obtained after text branching in the diffusion block.

[0230] The image branch is used to process the image representation #1 of the input image to obtain the image representation #3 of the input image.

[0231] The numbering of image representations #2 and #3 of the input image is for descriptive convenience only and has no limiting effect. For diffusion block #1, the image representation of the output diffusion block after image branching can be regarded as the image representation #2 of the input image, and the image representation #3 of the input image can be regarded as the intermediate result obtained after image branching in the diffusion block.

[0232] The noise branch can be used to process the noisy representation #1 based on the text representation #3 of the input text and the image representation #3 of the input image to obtain the noisy representation #2.

[0233] Furthermore, the noise branch may include sub-branch #1 and sub-branch #2. The diffusion block #1 may also include attention layer #1 and attention layer #2. Attention layer #1 may be considered to belong to sub-branch #1 or not. Attention layer #2 may be considered to belong to sub-branch #2 or not. The above schemes are merely described differently and do not constitute a limitation on the schemes of the embodiments of this application.

[0234] The numbers #1 and #2 for sub-branch are only used to distinguish the two sub-branches and do not have a limiting effect.

[0235] The numbers #1 and #2 for attention layers are only used to distinguish between the two attention layers and do not have a limiting effect.

[0236] Sub-branch #1 can be used to process the noisy representation #1 to obtain the noisy representation #3.

[0237] Noisy representations #2 and #3 are for descriptive convenience only and are not intended to limit the scope of the application. For diffuse block #1, the noisy representation output after noise branch processing can be considered as noisy representation #2. Noisy representation #3 is the intermediate result of sub-branch #1 processing noisy representation #1.

[0238] Attention layer #1 is used to perform attention operation #1 based on the text representation #3 and the noisy representation #3 of the input text to obtain operation result #1.

[0239] Alternatively, the text branch and sub-branch #1 can be viewed as sharing the attention layer #1.

[0240] As shown in Figure 5(a), the text representation #3 of the input text is obtained by processing the text representation #1 of the input text through the network layer located before the attention layer #1 in the text branch. The noisy representation #3 is obtained by processing the noisy representation #1 through the network layer located before the attention layer #1 in the sub-branch #1.

[0241] Sub-branch #2 can be used to process noisy representation #1 to obtain noisy representation #4.

[0242] Noisy representation #4 is the intermediate result of the process of sub-branch #2 processing noisy representation #1.

[0243] Attention layer #2 is used to perform attention operation #2 on the image representation #3 and the noisy representation #4 of the input image to obtain operation result #2.

[0244] Alternatively, the image branch and sub-branch #2 can be viewed as sharing the attention layer #2.

[0245] As shown in Figure 5(a), the image representation #3 of the input image is obtained by processing the image representation #1 of the input image through the network layer located before the attention layer #2 in the image branch. The noisy representation #4 is obtained by processing the noisy representation #1 through the network layer located before the attention layer #2 in the sub-branch #2.

[0246] The fusion result of operation result #1 and operation result #2 is used to determine the noisy characterization #2, or in other words, to determine the output of the diffusion block, or to update the noisy characterization.

[0247] "Merge" can also be replaced with other descriptive methods such as "merge," and can be achieved in various ways. For example, to merge operation result #1 and operation result #2, it can be done by adding operation result #1 and operation result #2 together, and the result of the sum can be used as the merged result.

[0248] As shown in Figure 5(a), the noise branch can be used to process the fusion result to obtain a noisy characterization #2.

[0249] Sub-branch #1 and sub-branch #2 can share the network layers after attention layer #1 and attention layer #2 in the noisy branch. The noisy representation #2 can be understood as being obtained by processing the fusion result by sub-branch #1 or sub-branch #2.

[0250] Furthermore, the output of attention layer #1 (such as operation result #1) can be used to determine the text representation #2 of the input text, and the output of attention layer #1 (such as operation result #2) can be used to determine the image representation #2 of the input image.

[0251] As shown in Figure 5(a), the text representation #2 can be obtained by processing the operation result #1 through the network layer located after the attention layer #1 in the text branch.

[0252] As shown in Figure 5(a), image representation #2 can be obtained by processing operation result #2 through a network layer located after attention layer #2 in the image branch.

[0253] The numbers #1 and #2 for attention operations are only used to distinguish the attention operations of the two attention layers and do not have any other limiting function.

[0254] The numbers #1 and #2 are used only to distinguish the outputs of the two attention layers and have no other limiting function.

[0255] The numbering of Noisy Representation #3 and Noisy Representation #4 is for descriptive convenience only and has no limiting effect. The representation of the text modality used as input to attention layer #1 is text representation #3, and the representation of the noise modality used as input to attention layer #1 is noisy representation #3. The representation of the image modality used as input to attention layer #2 is image representation #3, and the representation of the noise modality used as input to attention layer #2 is noisy representation #4.

[0256] Taking two diffusion blocks as an example, for instance, diffusion block #1 can be the first diffusion block, and the diffusion block following the first diffusion block is the second diffusion block. The first diffusion block and the second diffusion block have the same structure.

[0257] The noise branch, text branch, image branch, attention layer #1, attention layer #2, sub-branch #1, and sub-branch #2 in the first diffusion block are respectively called the first noise branch, the first text branch, the first image branch, the first attention layer, the second attention layer, the first sub-branch, and the second sub-branch.

[0258] Accordingly, the outputs of the first text branch, the first image branch, and the first noise branch can be used to determine the input of the second diffusion block. The specific processing steps can be found above and will not be repeated here.

[0259] Furthermore, in some scenarios, parts of the network layers in sub-branch #1 and sub-branch #2 may be shared. In this case, noisy representation #3 and noisy representation #4 may be identical.

[0260] In a diffusion block, the parameters of the three branches of the network layer can be set independently, which is equivalent to setting independent sub-models (i.e., branches) for the data of each modality. This allows the data of each modality to be processed independently, except for attention operations. When performing attention operations, the representations of the text modality and the noise modality are combined, and the representations of the image modality and the noise modality are combined, thereby realizing the update of the representations of each modality.

[0261] In the scheme of this application embodiment, during the processing of the diffusion block, the representation of the corresponding modality can be processed through multiple branches respectively. The text branch and the image branch can be regarded as two conditional branches of the noise branch (i.e., the main branch), or in other words, the text branch and the image branch share the noise branch. That is, the text representation and the image representation after being processed by the text branch and the image branch respectively can be used for the processing of noisy representation, or in other words, for denoising processing. For example, during the processing of the diffusion block, attention operation #1 is performed on the representation of the text modality and the representation of the noise modality to obtain operation result #1. Operation result #1 can be used to update the representation of the text modality. Attention operation #2 is performed on the representation of the image modality and the representation of the noise modality to obtain operation result #2. Operation result #2 can be used to update the representation of the image modality. Operation result #1 and operation result #2 are fused, and the fused result can be used to update the representation of the noise modality.

[0262] As one possible implementation, at least one of the text branch, noise branch, and image branch is based on a transformer block. A transformer block can also be called a transformer layer. A diffusion block can also be called an Ψ-DiT module.

[0263] Figure 5 shows a schematic diagram of the structure of a diffusion block. The text branch, noise branch, and image branch shown in Figure 5(b) are all constructed based on transformer blocks.

[0264] In the text branch, the data input to the text branch is processed sequentially through a layer normalization (LN) layer (LN 1-1 in the figure) and a linear layer (linear 1-1 in the figure), resulting in three outputs, corresponding to the query (Q), key (K), and value (V) of the attention layer (attention layer 1 in the figure). The three outputs of linear 1-1 can be used as an example of text representation #3. Attention layer 1 can be used as an example of attention layer #1. The data input to the text branch can be used as an example of text representation #1.

[0265] In the noise branch, the data input to the noise branch is processed by an LN layer (LN 2-1 in the figure), and then by two linear layers (linear 2-1 and linear 2-2 in the figure), each producing three outputs. The three outputs of one linear layer, linear 2-1, correspond to the Q, K, and V values ​​of an attention layer (attention 1 in the figure). The three outputs of linear 2-1 can serve as an example of noisy representation #3. The three outputs of the other linear layer, linear 2-2, correspond to the Q, K, and V values ​​of another attention layer (attention 2 in the figure). The three outputs of linear 2-2 can serve as an example of noisy representation #4. Attention layer 2 can serve as an example of attention layer #2. The data input to the noise branch can serve as an example of noisy representation #1.

[0266] The three outputs of linear 1-1 in the text branch and the three outputs of linear 2-1 in the noise branch are concatenated. The concatenated result is used as the Q, K, and V of attention layer 1 for attention operation. The result is then processed sequentially through the linear layer (linear 1-2 in the figure), the LN layer (LN 1-2 in the figure), and the multilayer perceptron (MLP) (MLP 1 in the figure). The result of MLP 1 is added to the input data to the text branch through residual connections to obtain the final processing result of the text branch. The processing result of attention layer 1 can be used as an example of operation result #1. The final processing result can be used as an example of text representation #2.

[0267] In the image branch, the data input to the image branch is processed sequentially through an LN layer (LN 3-1 in the figure) and a linear layer (linear 3-1 in the figure), resulting in three outputs, corresponding to Q, K, and V of the attention layer (attention layer 2 in the figure). The three outputs of linear 3-1 can be used as an example of image representation #3. The data input to the image branch can be used as an example of image representation #1.

[0268] The three outputs of linear 3-1 in the image branch and the three outputs of linear 2-2 in the noise branch are concatenated. The concatenated result is used as Q, K, and V of attention layer 2, and attention operation is performed. The result is then processed sequentially through the linear layer (linear 3-2 in the figure), the LN layer (LN 3-2 in the figure), and the MLP (MLP 3 in the figure) of the image branch. The processing result of MLP 3 is added to the data input to the image branch through residual connections to obtain the final processing result of the image branch. The processing result of attention layer 2 can be used as an example of operation result #2, and the final processing result can be used as an example of image representation #2.

[0269] The processing results of attention layer 1 and attention layer 2 are processed by two linear layers (linear2-3 and zero-linear in the figure) of the noise branch, respectively, and then fused. The fused result is then processed by the LN layer (LN 2-2 in the figure) and MLP (MLP 2 in the figure) of the noise branch. The processing result of MLP 2 is added to the data input to the noise branch through residual connections to obtain the final processing result of the noise branch. This final processing result can be used as an example of noisy representation #2. For example, as shown in Figure 5, the fusion method can be addition.

[0270] It should be understood that the structure of model 300 shown in Figure 3 is merely an example and does not constitute a limitation on the solutions of the embodiments of this application. For example, in the relevant description of Figure 3 above, the text processing module 310 and the image processing module 320 are regarded as sub-models of model 300. In other implementations, the text processing module 310 and / or the image processing module 320 can also be regarded as models independent of model 300. In addition, in other implementations, model 300 may also include other modules, which are not limited in the embodiments of this application.

[0271] According to the scheme of this application embodiment, attention operation is used to combine text information and image information to update the data of the noise mode. This allows for better reference to the information in the text space and image space to update the data of the noise mode, thereby improving the update effect of the noise mode representation and thus improving the image generation effect.

[0272] Furthermore, in this embodiment, each branch processes the data of its corresponding modality, so that the data of each modality can work in its own space. At the same time, the text branch and the image branch serve as two conditional branches of the noise branch. In this way, when the noise branch updates the representation of the noise modality, it can refer to the processing results of the text branch and the image branch, which is conducive to better balancing the injection of text information and image information, thereby improving the updating effect of the noise modality representation and thus improving the image generation effect.

[0273] Furthermore, in the solution of this application embodiment, a three-branch dual-condition injection structure is adopted, with the text branch and the image branch sharing the noise branch, resulting in a simple structure.

[0274] In addition, the three branches can process the representation of the corresponding modes in parallel, which is beneficial to improving the model's generation ability and convergence speed.

[0275] Furthermore, by updating the representation of each modality separately for each branch, the representations of each modality can operate within their respective spaces. Simultaneously, attention operations can incorporate information from other spaces, facilitating more effective updates to the representations of each modality. For example, the representation of the noise modality is updated based on the fusion of its attention operation results with the text modality representation and its attention operation results with the image modality representation. By referencing information from both the text and image spaces, it's easier to balance the injection of text and image information, thus improving the update performance of the noise modality representation.

[0276] It should be understood that in the above scheme, the attention layer is considered not to belong to any of the branches. That is, each branch can call the corresponding attention layer to perform attention operations during processing, and continue to perform subsequent operations based on the operation results. In other descriptions, the attention layer can also be considered to belong to the above branches. For example, attention layer #1 belongs to the text branch and sub-branch #1, and attention layer #2 belongs to the image branch and sub-branch #2. This scheme differs from the previous scheme only in the description method; the model structure is not fundamentally different.

[0277] Figure 6 illustrates a training method for an image generation model provided in an embodiment of this application. The structure of the image generation model (such as the initial image generation model) in the training method 600 shown in Figure 6 can adopt the structure of model 300 in Figure 3, and the relevant description of the model can be referred to the relevant description in Figure 3. To avoid repetition, some descriptions are appropriately omitted when describing the training method 600 shown in Figure 6. In method 600, the model that has not been trained is the initial image generation model.

[0278] For example, the method shown in Figure 6 can be executed by the system architecture shown in Figure 1 or Figure 2.

[0279] As shown in Figure 6, method 600 may include the following steps.

[0280] 610. Obtain the training dataset. The training dataset consists of multiple image-text pairs. Each image-text pair includes a matching sample image, a target sample image, and sample text. In an image-text pair, the resolution of the sample image is smaller than that of the target sample image. The sample text can be used to provide hints about the content of the image to be generated.

[0281] 620. The initial image generation model is trained using the training dataset to obtain a trained image generation model. The image generation model can be used to iteratively denoise the initial noisy representation at multiple time steps based on the image and text input to the model, thereby obtaining the generated image.

[0282] The training data will be explained below.

[0283] Each image-text pair can include a sample image, a target sample image, and sample text that have a matching relationship. In the two images of an image-text pair, the image with lower resolution can be regarded as the sample image, and the image with higher resolution can be regarded as the target sample image.

[0284] As mentioned earlier, image generation models can be used to perform super-resolution tasks. Accordingly, in a text-image pair, the sample image and the target sample image can be images of different resolutions for the same scene, that is, the content of the sample image and the target sample image can be the same.

[0285] For ease of description, in the embodiments of this application, the image with lower resolution (i.e., the sample image) in an image pair is called a low-resolution image, and the image with higher resolution (i.e., the target sample image) is called a high-resolution image; the two are corresponding. For example, a low-resolution image can be obtained by downsampling the corresponding high-resolution image. The high-resolution image serves as the reference target for the corresponding low-resolution image during model training.

[0286] For example, in a text-image pair, the sample text can be the caption of a low-resolution image, or it can be the caption of a high-resolution image.

[0287] The training dataset can be derived from a public dataset, and correspondingly, the descriptive text in the image-text pair can also be derived from a public dataset. In this case, the descriptive text obtained from the public dataset can be directly used as sample text, or the descriptive text obtained from the public dataset can be processed (such as error correction, filtering, etc.) before being used as sample text. This application does not limit this approach.

[0288] Alternatively, the training dataset can be constructed by yourself.

[0289] As an example, sample text can be a low-resolution image or high-resolution descriptive text.

[0290] For example, the descriptive text can be generated by a descriptive text generation model.

[0291] Low-resolution or high-resolution images are input into a descriptive text generation model for processing to generate descriptive text, which can be used as sample text.

[0292] The descriptive text generation model can also be replaced by a captioner model or other descriptive methods. For example, the descriptive text generation model can be a multi-modal large language model. As shown in Figure 7, a high-resolution image is input into the multi-modal large language model using a prompt template for processing to generate descriptive text.

[0293] The text generation model can be part of the initial image generation model or it can be a model independent of the initial image generation model. This application does not limit this.

[0294] The above are just examples. The descriptive text can also be obtained in other ways, such as by writing it manually.

[0295] As another example, sample text can be an instruction (or prompt) used to guide the initial image generation model's task. For example, sample text could be "Please generate an image of... based on the input image."

[0296] The low-resolution and high-resolution images in an image-text pair can be from publicly available datasets or be custom-constructed. For example, the low-resolution image can be obtained by downsampling the corresponding high-resolution image.

[0297] The above are merely examples and do not constitute a limitation on the solutions implemented in this application.

[0298] The training process of the initial image generation model is explained below.

[0299] Similar to the diffusion model, the training of the initial image generation model consists of two phases: a forward process (noise diffusion) and a backward process (denoising and restoration).

[0300] The forward process, also known as the forward diffusion process, is essentially a noise addition process. In this process, starting with the real data x0, noise is gradually added over T time steps, ultimately yielding the noisy result x. TThe noise addition method for each time step can refer to the noise addition method of the diffusion model, and will not be repeated here.

[0301] The actual data x0 represents the true noisy representation at time step 0. The noisy result x... T This represents the true noisy representation of time step T. When t' is greater than 0, the true noisy representation of time step t' is obtained by adding noise to the true noisy representation of time step t'-1. The added noise is the true noise of time step t', and the true noisy representation of time step t' is the true noise result of time step t'-1. Correspondingly, the true noisy representation of time step t'-1 is the true denoising result of time step t'. T' is a positive integer.

[0302] It should be understood that in the embodiments of this application, the noisy representation of each time step is for descriptive convenience only, used to describe a noise-related representation, and does not imply that noise is necessarily added to the noisy representation of each time step. For example, at time step 0, no noise has been added, therefore, the true noisy representation of time step 0 does not contain added noise, and can also be called the true representation of time step 0.

[0303] The true noisy representation at time step 0 is the image representation of the high-resolution image.

[0304] In method 600, the format of the noisy representation depends on the requirements of the diffusion module 330 for the input format. An example of the noisy representation at time step 0 is provided below.

[0305] For example, the true noisy representation of time step 0 can be the high-resolution image itself.

[0306] For example, the true noisy representation at time step 0 can be a latent representation of the high-resolution image. For instance, the latent representation of the high-resolution image can be obtained by encoding the high-resolution image using a VAE encoder.

[0307] For example, the true noisy representation of time step 0 can be obtained by tokenizing a high-resolution image or a potential representation of a high-resolution image.

[0308] The reverse process, also known as the reverse generation process, reverse diffusion process, sampling process, or denoising process, starts with noise and gradually removes it to recover the true data. Given a noisy representation at a given time step, the initial image generation model can determine the predicted noise and / or denoising result at that time step. Using the predictions at each time step, the noisy representation at time step T can be gradually recovered to the noisy representation at time step 0. The predicted noise is the noise predicted by the diffusion module 330. The denoising result is the denoising result predicted by the diffusion module 330. This process needs to be learned so that the initial image generation model can learn how to remove noise at each time step. That is, training the diffusion module 330 to learn how to accurately predict the noise and / or denoising result at each time step enables it to gradually denoise and generate new samples that approximate the true data distribution given random noise.

[0309] Step 620 may include:

[0310] During an iteration, the initial image generation model determines the predicted noise and / or the denoising result of time step t' based on the initial image representation of the sample image, the initial text representation of the sample text, and the true noisy representation of time step t'.

[0311] The parameters of the initial image generation model were adjusted with the goal of reducing diffusion loss.

[0312] The diffusion loss includes the difference between the predicted noise at time step t' and the actual noise at time step t', and / or the difference between the denoising result at time step t' and the actual noisy representation at time step t'+1.

[0313] The true noisy representation at time step t' is obtained by adding noise to the image representation of the target sample image. t' is a positive integer. The true noisy representation at time step t'+1 is obtained by adding noise to the image representation of the target sample image.

[0314] The actual noise at time step t' is the noise added at time step t' during the forward process. For a detailed description, please refer to the description of the forward process above.

[0315] For example, taking the initial image generation model used to determine the prediction noise as an example, during the training process, time step t' can be randomly sampled. The initial image generation model determines the prediction noise at time step t' based on the initial image representation of the sample image, the initial text representation of the sample text, and the real noisy representation at time step t'. The diffusion loss is calculated based on the prediction noise at time step t' and the real noise at time step t', and the parameters of the initial image generation model are updated based on the diffusion loss.

[0316] The above process can be understood as follows: the training objective of the model is to predict the real noise added at time step t' as accurately as possible.

[0317] The predicted noise in the above process can also be replaced with the denoised result, and the corresponding description can be adjusted adaptively, which will not be elaborated here.

[0318] The structure of the initial image generation model can adopt the structure of model 300 shown in Figure 3. For related descriptions, please refer to the description of model 300 above. "Input text" can be replaced with "sample text", and "input image" can be replaced with "sample image".

[0319] For example, the specific description of the initial image representation of the sample image can refer to the description of the initial image representation of the input image of Model 300 above, and the input image adaptability can be replaced with the sample image.

[0320] For example, the specific description of the initial text representation of the sample text can refer to the previous description of the initial text representation of the input text of Model 300, and the input text adaptability can be replaced with the sample text.

[0321] After training, during the inference phase, the model can gradually remove noise from pure noise to generate new data. For example, at a given time step t' in the backdiffusion process, the diffusion model can predict noise, which can be used to update the noisy representation at time step t' to move to the next time step t'-1, thus obtaining the noisy representation at time step t'-1.

[0322] Optionally, step 620 may include: performing an attention operation #61 on the text representation #61 (i.e., the first text representation of the sample text) and the noisy representation #61 (i.e., the fourth noisy representation) of the sample text to obtain an operation result #61 (i.e., the third operation result); performing an attention operation #62 on the image representation #61 (i.e., the first image representation of the sample image) and the noisy representation #62 (i.e., the fifth noisy representation) of the sample image to obtain an operation result #62 (i.e., the fourth operation result); determining the predicted noise at time step t' and / or the denoising result at time step t' based on the fusion result of operation result #61 and operation result #62.

[0323] The text representation #61 of the sample text is determined based on the initial text representation of the sample text. The image representation #61 of the sample image is determined based on the initial image representation of the sample image. The noisy representation #61 and the noisy representation #62 are determined based on the true noisy representation at time step t'.

[0324] The numbers #61 for text representation, #61 for noisy representation, #61 for operation result, #61 for image representation, #62 for noisy representation, #62 for operation result, #61 for attention operation, and #62 for attention operation are for descriptive convenience only and do not have any limiting effect.

[0325] The detailed description of the processing procedure of the initial image generation model described above can be found in the relevant description of Model 300. Specifically, the detailed descriptions of text representation #61, image representation #61, noisy representation #61, noisy representation #62, operation result #61, operation result #62, attention operation #61, and attention operation #62 can be found in the relevant descriptions of text representation #3 of input text, image representation #3 of input image, noisy representation #3, noisy representation #4, operation result #1, operation result #2, attention operation #1, and attention operation #2, respectively. The only difference is that noisy representation #61 and noisy representation #62 can be obtained by processing the real noisy representation at time step t'.

[0326] Optionally, the initial image generation model includes a text branch #61 (i.e., the first text branch during training), a noise branch #61 (i.e., the first noise branch during training), an image branch #61 (i.e., the first image branch during training), an attention layer #61 (i.e., the first attention layer during training), and an attention layer #62 (i.e., the second attention layer during training).

[0327] The text representation #61 of the sample text is obtained by processing the initial text representation of the sample text through text branch #61.

[0328] The image representation #61 of the sample image is obtained by processing the initial image representation of the sample image through image branch #61.

[0329] The noisy representation #61 is obtained by processing the real noisy representation of time step t' through sub-branch #61 of the noise branch #61 (i.e., the first sub-branch of the training phase).

[0330] The noisy representation #62 is obtained by processing the real noisy representation of time step t' through sub-branch #62 of the noise branch #61 (i.e., the second sub-branch of the training phase).

[0331] The operation result #61 is obtained by performing attention operations #61 on the text representation #61 and the noisy representation #61 of the sample text through the attention layer #61.

[0332] Operation result #62 is obtained by performing attention operation #62 on the image representation #61 and the noisy representation #62 of the sample image through attention layer #62.

[0333] The numbers #61 for text branch, #61 for noise branch, #61 for image branch, #61 for attention layer, #62 for attention layer, #61 for sub-branch, and #62 for sub-branch are for descriptive convenience only and do not have any limiting effect.

[0334] Optionally, at least one of the text branch #61, image branch #61, and noise branch #61 can be constructed based on the transformation transformer layer.

[0335] As shown in Model 300, the diffusion module of the initial image generation model may include one or more diffusion blocks. Each diffusion block may include a text branch, a noise branch, and an image branch. For example, the one or more diffusion blocks may include diffusion block #61 (i.e., the first diffusion block in the training phase). Text branch #61, noise branch #61, image branch #61, attention layer #61, and attention layer #62 belong to diffusion block #61. Diffusion block #61 can be regarded as the first diffusion block in the diffusion module. A detailed description of the diffusion blocks can be found in Model 300, and will not be repeated here.

[0336] Furthermore, the one or more diffusion blocks may also include diffusion block #62 (i.e., the second diffusion block during the training phase). The structure of diffusion block #62 is the same as that of diffusion block #61.

[0337] The output of diffusion block #61 can be used as the input of diffusion block #62, or used to determine the input of diffusion block #62.

[0338] Optionally, the input to the diffusion block #62 may include the text representation #62 of the sample text (i.e., the second text feature of the sample text), the image representation #62 of the sample image (i.e., the second image representation of the sample image), and the noisy representation #63 (i.e., the sixth noisy representation). The text representation #62 of the sample text is obtained by processing the operation result #61 through the text branch #61; the image representation #62 of the sample image is obtained by processing the operation result #62 through the image branch #61; and the noisy representation #63 is obtained by processing the fusion result of the operation result #61 and the operation result #62 through the noise branch #61.

[0339] The numbering of text representation #62, image representation #62, and noisy representation #63 is for descriptive convenience only and has no limiting effect.

[0340] Diffusion block #62 and diffusion block #61 can be the first and second diffusion blocks in model 300. The specific processing procedures, inputs and outputs of each branch can be referred to the relevant descriptions in model 300 above. To avoid repetition, they will not be repeated here.

[0341] Further, optionally, during training, the initial image generation model is built based on the pre-trained model, and step 620 may include: adjusting other parameters in the initial image generation model besides the parameters of the pre-trained model with the goal of reducing diffusion loss, and freezing the parameters of the pre-trained model.

[0342] For example, obtain a pre-trained text-to-image model, build an initial image generation model based on the pre-trained text-to-image model, and train it.

[0343] The text-to-image (T2I) model refers to a model used to achieve the task of generating text-to-image (T2I) (i.e., the text-to-image task).

[0344] The pre-trained model can be a DiT model. This DiT model can be a text-to-image model. For example, the pre-trained model can adopt a structure similar to the initial image generation model. Specifically, the difference between the two structures is that a diffusion block of the pre-trained model only includes a text branch, an attention layer, and a sub-branch of the noise branch.

[0345] After obtaining the pre-trained model, another sub-branch of the image branch and noise branch can be constructed in each diffusion block to obtain the initial image generation model and train it.

[0346] In this case, the model architecture in the embodiments of this application can also be understood as a three-branch fine-tuning architecture based on a pre-trained T2I model, such as Ψ-DiT.

[0347] Taking diffusion block #61 as an example, text branch #61, sub-branch #61 and attention layer #61 belong to the pre-trained model, while image branch #61, sub-branch #62 and attention layer #62 do not belong to the pre-trained model.

[0348] In this case, during training, for the diffusion block #61, only the parameters of image branch #61, sub-branch #62, and attention layer #62 are updated.

[0349] Furthermore, sub-branch #61 and sub-branch #62 may share network layers, and the parameters of the shared network layers can be frozen. That is, during training, for the diffusion block #61, only the parameters in sub-branch #62 that are not shared with sub-branch #61, the parameters of attention layer #62, and the parameters of image branch #62 are updated.

[0350] Figure 8 shows a schematic diagram of a diffusion block in a pre-trained model. As shown in Figure 8, the diffusion block of the pre-trained model only includes a sub-branch of the text branch and the noise branch.

[0351] For example, as shown in Figure 8, sub-branch #61 of the noise branch #61 in the diffusion block #61 includes LN 2-1, linear 2-1, linear 2-3, LN 2-2, MLP 2, and residual connections. Sub-branch #62 of the noise branch #61 includes LN 2-1, linear 2-2, zero-linear, LN 2-2, MLP 2, and residual connections. Sub-branch #1 and sub-branch #2 share LN 2-1, LN 2-2, MLP 2, and residual connections. During training, the parameters of the text branch #61, attention layer #61 (i.e., attention 1), sub-branch #61, and the parts of sub-branch #62 shared with sub-branch #61 are frozen, while the parameters of the remaining parts are updated.

[0352] In this way, the generative priors of the pre-trained model can be fully utilized. For example, the generative priors of the text-to-image model (such as text information) can be used to further improve the performance of the image SR task and improve the quality of the generated images. At the same time, fine-tuning can be performed on the pre-trained model, which can help improve training efficiency.

[0353] Furthermore, the parameters of image branch #61 are initialized based on the parameters of sub-branch #61 of noise branch #61.

[0354] In other words, the parameters of image branch #61 are initialized based on the parameters of sub-branch #61.

[0355] The parameters of sub-branch #61 are obtained through pre-training, which makes full use of the prior knowledge of the pre-trained model, which is beneficial to improving the convergence speed and generation effect of the model.

[0356] Furthermore, the parameters of the output layer of attention layer #62 are initialized with zero initialization.

[0357] This helps ensure a seamless transition from text-to-image tasks to image-to-image tasks.

[0358] It should be understood that the above explanation uses only one diffusion block (i.e., diffusion block #61) as an example. The description of other diffusion blocks can be found in diffusion block #61. To avoid repetition, it will not be repeated here.

[0359] Furthermore, a MIM strategy can be introduced during the training process.

[0360] Specifically, the initial image representation of the sample image can be obtained by performing a masking operation on the sample image.

[0361] In other words, a masking operation needs to be performed during the initial image representation of the sample image.

[0362] For example, a masking operation is performed on multiple patches of a sample image to obtain a masking result, based on which an initial image representation of the sample image can be obtained.

[0363] Method 600 may further include: performing a masking operation on the sample image to obtain a masking result; and generating an initial image representation of the sample image based on the masking result.

[0364] During training, the diffusion loss and MIM loss are optimized. The MIM loss can be used to indicate the difference between the predicted masked content and the true masked content.

[0365] Taking the diffusion module operating in the latent space as an example, the dimension of the initial image representation is the dimension of the latent space. For example, a sample image is encoded to obtain its latent representation. The latent representation of the sample image is then segmented into multiple patches. A masking operation is performed on a portion of these patches, and the unmasked patches are flattened to form a token sequence. Each token in the token sequence is then mapped to a vector space, and positional information is added to each token to obtain the image embedding.

[0366] Masking operations can be directly applied to the multiple patches obtained after splitting.

[0367] Alternatively, as shown in Figure 7, the multiple patches obtained after segmentation can be flattened, a masking operation can be performed on the flattened patches, and the unmasked patches after the masking operation can be flattened to form a token sequence. Then, each token in the token sequence is mapped to a vector space, and positional information is added to each token to obtain an image embedding. The positional information of each token is still its position in two-dimensional space. It should be understood that the positions of the patches in two-dimensional space after the masking operation in the figure are only illustrative and do not imply that the multiple patches must be restored to their positions in two-dimensional space after the masking operation.

[0368] The above are just examples. In other implementations, masking operations can also be applied to other dimensions. For example, if the diffusion module operates in pixel space, that is, the dimension of the initial image representation is the dimension of pixel space, then the masking operation can be applied to multiple patches obtained by segmenting the sample image.

[0369] For example, the masked patch can be randomly determined. Alternatively, the masked patch can be determined in other ways, which is not limited in this embodiment.

[0370] The mask ratio can be determined in a variety of ways.

[0371] The masking ratio can be the ratio of the number of masked patches to the number of patches in the sample image.

[0372] For example, the mask ratio can be a pre-set fixed value.

[0373] For example, the mask ratio can be a random value.

[0374] If the mask ratio is determined in the above way, the learning process for the network may become quite difficult.

[0375] In view of this, embodiments of this application provide a progressive MIM strategy.

[0376] Optionally, the mask ratio is related to the number of training steps.

[0377] The number of training steps can also be replaced with the number of iterations.

[0378] Optionally, the masking ratio used in the p-th iteration of the training process is negatively correlated with the value of p, where p is a positive integer.

[0379] For example, in the p1th iteration, the mask ratio is a, and in the p2th iteration, the mask ratio is b. p1 and p2 are positive integers, p2 is greater than p1, and a is greater than b.

[0380] p1 and p2 can be any two iterations during the training process. The mask ratio in the later iteration can be smaller than the mask ratio in the earlier iteration.

[0381] In other words, in the early stages of training, the mask ratio is relatively large, and less image information is injected into the model. As training progresses, the mask ratio gradually decreases, and the image information injected into the model gradually increases.

[0382] For example, further, when p is greater than or equal to threshold #1 (i.e., the first threshold), the mask ratio is ratio #1 (i.e., the first ratio). When p is less than threshold #1, the mask ratio used to perform the masking operation in the p-th iteration is negatively correlated with the value of p. This ratio #1 can be a fixed value r. min r min Greater than or equal to 0.

[0383] In other words, as training progresses to a certain stage, the mask ratio can be set to a fixed value and will no longer decrease.

[0384] Threshold #1 can be set as needed.

[0385] For example, threshold #1 can be determined based on the total number of steps in the current training phase. For instance, threshold #1 could be 1 / 3 of the total number of steps.

[0386] Alternatively, the threshold #1 can also be a pre-set fixed value, which is not limited in this embodiment of the application.

[0387] For example, when the current training steps are less than the threshold #1, the mask ratio is negatively correlated with the number of training steps. That is, as the current training steps increase, the mask ratio gradually decreases.

[0388] For example, the mask ratio r can satisfy the following formula:

[0389] Where p represents the current training step number. c represents the asymptotic step number, which can be a pre-set fixed value. k represents the threshold #1. σ represents an adjustable constant that controls the rate at which the mask ratio changes.

[0390] In the scheme of this application embodiment, a progressive MIM strategy is adopted for training, dynamically adjusting the mask ratio to smoothly migrate the model from a text-based image model to an image-based image model. In other words, optimizing the transition from text-based to image-based tasks improves training speed and effectiveness, while enhancing the model's generalization ability to low-resolution degradation. Specifically, in the early stages of training, a larger mask ratio is used to make the initial image generation model training process closer to the pre-training process, such as the text-based image training paradigm, thereby reducing the difficulty of network learning. In the later stages of training, after the model gradually adapts to the injection of low-resolution images, a lower mask ratio is used so that low-resolution images can provide more information, further exploring the model's image-based generation capabilities. This allows for training the mask-based model in a progressive manner, optimizing the model's transition ability, reducing the overall difficulty of model training, thereby reducing training costs, improving training speed and effectiveness, and enhancing the model's generalization ability to low-resolution degradation.

[0391] Figure 7 illustrates a schematic diagram of an image processing procedure. It should be understood that Figure 7 is a simplified schematic diagram, omitting some modules, such as the text processing module.

[0392] As shown in Figure 7, the prompt consisting of k demonstration examples and a new query is input into the multimodal large language model, which then generates the response corresponding to the new query. Specifically, as shown in Figure 7, taking a demonstration example, for instance, user:[sample_img_1,question], assistant:[sample_answer_1], where "user" represents the user, "[sample_img_1,question]" represents the user's query, sample_img_1 represents an image sample, and question represents the question about that image sample, such as "Please generate a descriptive text." "Assistant" instructs the multimodal large language model, and [sample_answer_1] represents an answer sample, which is the expected answer output by the multimodal large language model. Other demonstration examples follow the same logic. In the new query, HR_image represents a high-resolution image, i.e., the target sample image. The multimodal large language model generates descriptive text for the high-resolution image.

[0393] As shown in Figure 7, the low-resolution image, i.e., the sample image, is input into the VAE encoder for encoding to obtain the latent representation. The latent representation is then processed using a progressive masking method, and the representation input to the diffusion module is generated based on the masking result. The specific processing procedure can be found in the previous text and will not be repeated here.

[0394] The embedded representation of the descriptive text and the embedded representation of the masked image are input into the diffusion module, which performs iterative denoising. The final denoising result is then restored to the pixel space by the VAE decoder in order to generate a high-resolution image.

[0395] Figure 9 illustrates an image generation method provided by an embodiment of this application.

[0396] For example, the method 900 shown in FIG9 can be executed by the system architecture shown in FIG1 or FIG2.

[0397] For example, if a user needs to improve the quality of an image (e.g., increase the resolution), taking the system shown in Figure 1(a) as an example, the user equipment (e.g., a client device) can receive the image input by the user and send the image to a data processing device (e.g., a server device). The data processing device executes method 900 to generate an image and returns it to the user equipment. The user equipment can then present the generated image to the user.

[0398] The image generation model in method 900 can adopt the structure of model 300 in Figure 3, and the relevant description of the model can be found in the relevant description in Figure 3. To avoid repetition, some descriptions are omitted when describing method 900 shown in Figure 9.

[0399] For example, the image generation model in method 900 can be obtained by method 600.

[0400] As shown in Figure 9, method 900 may include the following steps.

[0401] 910, Get image #1 (i.e., the first image).

[0402] 920, Get text #1 (i.e., the first text), text #1 is used to indicate the content of the image #2 to be generated.

[0403] 930. Based on text #1 and image #1, the initial noisy representation is iteratively denoised at multiple time steps to generate image #2 (i.e., the second image).

[0404] Image #2 has a higher resolution than image #1.

[0405] The designation of image #1 is for descriptive convenience only and does not constitute a limitation on the embodiments of this application. For example, when method 900 is applied to a super-resolution task, image #1 can be any image that needs to be super-resolution processed.

[0406] In step 910, image #1 can be obtained in a variety of ways.

[0407] For example, step 910 may include: receiving image #1 input by the user, or receiving image #1 sent by another device, or reading image #1 stored locally, etc.

[0408] The above are merely examples and do not constitute a limitation on the solutions implemented in this application.

[0409] The number #1 in the text is for convenience of description only and does not constitute a limitation on the solutions of the embodiments of this application.

[0410] As an example, text #1 could be the descriptive text for image #1.

[0411] The descriptive text for image #1 can be obtained in several ways.

[0412] For example, the descriptive text can be generated by a descriptive text generation model.

[0413] Image #1 is input into the descriptive text generation model for processing to generate descriptive text, which can be used as text #1.

[0414] The description of the text generation model can be found in Method 600, simply by replacing the high-resolution image with image #1; further details are omitted here.

[0415] The text generation model can be part of the image generation model or it can be a model independent of the image generation model. This application does not limit this.

[0416] For example, the descriptive text can be entered by the user.

[0417] As another example, text #1 could be an instruction (or prompt) used to instruct the image generation model on its task. For example, text #1 could be "Please generate an image of... based on the input image".

[0418] The above are merely examples and do not constitute a limitation on the solutions implemented in this application.

[0419] Step 930 can be achieved using an image generation model. The processing procedure of the image generation model is explained below.

[0420] In step 930, the initial noisy representation is iteratively denoised at multiple time steps based on the initial text representation of text #1 and the initial image representation of image #1 to generate image #2.

[0421] For ease of description, in Method 900, it is assumed that the denoising process has a total of T time steps, i.e., T time points. From time step T to time step 0, the image generation model continuously performs iterative denoising processing to obtain the denoising results at each time step. The initial noisy representation is the noisy representation at time step T. The final denoising result is the noisy feature at time step 0.

[0422] Taking time step t as an example, during the denoising process of the noisy representation at time step t, the initial image representation of image #1, the initial text representation of text #1, and the noisy representation at time step t are input into the image generation model for processing to determine the denoising result at time step t. The noisy representation at time step t can be the initial noisy representation or the denoising result at time step t+1. t is a positive integer.

[0423] When t = T, the noisy representation of time step t is the noisy representation of time step T, which is the initial noisy representation.

[0424] When t < T, the noisy characterization of time step t is the denoising result of the previous time step (i.e., time step t+1).

[0425] The structure of the image generation model can adopt the structure of model 300 shown in Figure 3. For related descriptions, please refer to the description of model 300 above. "Input text" can be replaced with "text #1" and "input image" can be replaced with "image #1".

[0426] For example, the specific description of the initial image representation of image #1 can refer to the description of the initial image representation of the input image of model 300 above, and the input image adaptability can be replaced with image #1.

[0427] For example, the specific description of the initial text representation of text #1 can refer to the previous description of the initial text representation of the input text of model 300, and simply replace the input text adaptability with text #1.

[0428] Optionally, step 920 may include: during the denoising process of the noisy representation of time step t,

[0429] Perform attention operation #91 on the text representation #91 of text #1 (i.e., the first text representation of the first text) and the noisy representation #91 (i.e., the first noisy representation) to obtain operation result #91 (i.e., the first operation result);

[0430] Attention operation #92 is performed on the image representation #91 (i.e., the first image representation of the first image) and the noisy representation #92 (i.e., the second noisy representation) of image #1 to obtain operation result #92 (i.e., the second operation result). The fusion result of operation result #91 and operation result #92 is used to determine the denoising result at time step t. Noisy representation #91 and noisy representation #92 are obtained by processing the noisy representation at time step t. The denoising result at time step t is used to determine image #92. The noisy representation at time step t can be the initial noisy representation, or the noisy representation at time step t can be obtained by denoising the initial noisy representation. For example, when t = T, the noisy representation at time step t can be the initial noisy representation. When t ≠ T, the noisy representation at time step t can be obtained by iteratively denoising the initial noisy representation from time step T to time step t+1.

[0431] The text representation #91 of text #1 is determined based on the initial text representation of text #1, and the image representation #91 of image #1 is determined based on the initial image representation of image #1.

[0432] The text representation #91 of text #1, the image representation #91 of image #1, the noisy representation #91, the noisy representation #92, the operation result #91, the operation result #92 number, the attention operation #91, and the attention operation #92 are for descriptive convenience only and have no limiting effect.

[0433] For a detailed description of the processing steps of the above image generation model, please refer to the relevant description of Model 300. Specifically, for the detailed descriptions of the text representation #91 of text #1, the image representation #91 of image #1, the noisy representation #91, the noisy representation #92, the operation result #91, the operation result #92, the attention operation #91, and the attention operation #92, please refer to the relevant descriptions of the text representation #3 of input text, the image representation #3 of input image, the noisy representation #3, the noisy representation #4, the operation result #1, the operation result #2, the attention operation #1, and the attention operation #2, respectively.

[0434] Optionally, the image generation model includes a text branch #91 (i.e., the first text branch in the inference stage), an image branch #91 (i.e., the first image branch in the inference stage), a noise branch #91 (i.e., the first noise branch in the inference stage), an attention layer #91 (i.e., the first attention layer in the inference stage), and an attention layer #92 (i.e., the second attention layer in the inference stage).

[0435] The text representation #91 of text #1 is obtained by processing the initial text representation of text #1 through text branch #91.

[0436] Image representation #91 of image #1 is obtained by processing the initial image representation of image #1 through image branch #91.

[0437] The noisy representation #91 is obtained by processing the noisy representation of time step t through sub-branch #91 of the noise branch #91 (i.e., the first sub-branch of the inference stage).

[0438] The noisy representation #92 is obtained by processing the noisy representation of time step t through sub-branch #92 of the noise branch #91 (i.e., the second sub-branch of the inference stage).

[0439] Operation result #91 was obtained by performing attention operation #91 on the text representation #91 and the noisy representation #91 of text #1 through attention layer #91.

[0440] The result #92 is obtained by performing attention operation #92 on the image representation #91 and the noisy representation #92 of image #1 through attention layer #92.

[0441] The numbers #91 for text branch, #91 for noise branch, #91 for image branch, #91 for attention layer, #92 for attention layer, #91 for sub-branch, and #92 for sub-branch are for descriptive convenience only and do not have any limiting effect.

[0442] Optionally, at least one of the text branch #91, image branch #91, and noise branch #91 can be constructed based on the transformation transformer layer.

[0443] As shown in Model 300, the diffusion module of the image generation model may include one or more diffusion blocks. Each diffusion block may include a text branch, a noise branch, and an image branch. For example, the one or more diffusion blocks may include diffusion block #91 (i.e., the first diffusion block in the inference phase). Text branch #91, noise branch #91, image branch #91, attention layer #91, and attention layer #92 belong to diffusion block #91. Diffusion block #91 can be regarded as the first diffusion block in the diffusion module. A detailed description of the diffusion blocks can be found in Model 300, and will not be repeated here.

[0444] Furthermore, the one or more diffusion blocks may also include diffusion block #92 (i.e., the second diffusion block in the inference phase). Diffusion block #91 and diffusion block #92 have the same structure.

[0445] The output of diffusion block #91 can be used as the input of diffusion block #92, or used to determine the input of diffusion block #92.

[0446] Optionally, during the denoising process of the noisy representation at time step t, the input of the diffusion block #92 includes the text representation #92 of text #1 (i.e., the second text representation of the first text), the image representation #92 of image #1 (i.e., the second image representation of the first image), and the noisy representation #93 (i.e., the third noisy representation).

[0447] The text representation #92 of text #1 is obtained by processing the operation result #91 through text branch #91.

[0448] Image representation #92 of image #1 is obtained by processing operation result #92 through image branch #91.

[0449] The noisy characterization #93 is obtained by processing the fusion result of operation result #91 and operation result #92 through the noise branch #91.

[0450] The numbering of text representation #92 for text #1, image representation #92 for image #1, and noisy representation #93 is for descriptive convenience only and has no limiting effect.

[0451] Diffusion block #92 and diffusion block #91 can be the first and second diffusion blocks in model 300. The specific processing procedures, inputs and outputs of each branch can be referred to the relevant descriptions in model 300 above. To avoid repetition, they will not be repeated here.

[0452] The training process of the image generation model will be explained below.

[0453] Optionally, the image generation model is obtained by training the initial image generation model based on the training dataset. The training dataset includes multiple image-text pairs, which include sample images, target sample images, and sample text that have a matching relationship. The resolution of the sample images is smaller than that of the target sample images.

[0454] Optionally, the image generation model is obtained by adjusting the parameters of the initial image generation model with the goal of reducing the diffusion loss. The diffusion loss includes the difference between the predicted noise and the true noise at time step t', and / or the difference between the denoising result at time step t' and the true noisy representation at time step t'+1. Time step t' belongs to multiple time steps, and the true noisy representation at time step t' is obtained by adding noise to the image representation of the target sample image. t' is a positive integer, t'+1 belongs to multiple time steps, and the true noisy representation at time step t'+1 is obtained by adding noise to the image representation of the target sample image. Alternatively, the true noisy representation at time step t'+1 is the image representation of the target sample image, and the predicted noise and / or the denoising result at time step t' are obtained by processing the true noisy representation at time step t' using the initial image generation model based on the initial image representation of the sample image and the initial text representation of the sample text.

[0455] Optionally, the initial image generation model is built based on a pre-trained model. The image generation model is obtained by adjusting the parameters of the initial image generation model other than the parameters of the pre-trained model with the goal of reducing diffusion loss.

[0456] The pre-trained model is used to perform text-to-image tasks.

[0457] Optionally, text branch #91, sub-branch #91, and attention layer #91 belong to the pre-trained model, while image branch #91, sub-branch #92, and attention layer #92 do not belong to the pre-trained model.

[0458] Text branch #91, sub-branch #91, attention layer #91, image branch #91, sub-branch #92, and attention layer #92 correspond to text branch #61, sub-branch #61, attention layer #61, image branch #61, sub-branch #62, and attention layer #62 in method 600, respectively.

[0459] Optionally, in the p-th iteration of the training process, the initial image representation of the sample image is generated based on the masking result, which is obtained by performing a masking operation on the sample image. The masking ratio used in the masking operation is negatively correlated with the value of p, where p is a positive integer.

[0460] For a detailed description of the training process of the image generation model, please refer to Method 600. To avoid repetition, it will not be repeated here.

[0461] Table 1 shows the comparison results of relevant metrics for the two approaches. As shown in Table 1, compared with ControlNet methods, such as the ControlNet-based DiT architecture (DiT-controlnet), the three-branch DiT architecture of this application performs better and converges faster. Relevant metrics include: best iterations, peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), learned perceptual image patch similarity (LPIPS), manual image quality assessment (ManIQA), clip-based image quality assessment (ClipIQA), and multi-scale image quality (MUSIQ).

[0462] Table 1

[0463] Table 2 shows the test results of the scheme of the present application embodiment on multiple public test sets (DIV2K-Val, RealWorld60, and NTIRE2024-RAM50). As shown in Table 2, the scheme of the present application embodiment achieves the best overall performance on multiple public test sets. The schemes compared include: stable super-resolution (StableSR), blind image restoration with generative diffusion (DiffBIR), pixel-aware stable diffusion (PASD), semantics-aware real-world image super-resolution (SeeSR), scaling-up image restoration (SUPIR), and edge-aware module (EAM).

[0464] Table 2

[0465] Table 3 shows the comparison results of the progressive masked image training strategy and other training strategies in this application on the public test set. As shown in Table 3, compared with other training strategies, the progressive masked image training strategy significantly improves the objective index of the super-resolution image on the public test set of the blind super-resolution task.

[0466] Table 3

[0467] The solution proposed in this application can also be applied to video SR reconstruction. When used for video SR reconstruction, the solution can be adaptively adjusted. For example, time series information can be introduced, and the attention module in Figure 3 can be adjusted to an attention module that can extract spatial and temporal features. Time series analysis can be introduced to capture spatiotemporal information and enhance the model's ability to process dynamic images.

[0468] When performing super-resolution tasks on images within a specific domain, domain knowledge and prior information can be combined to further optimize the model's generative capabilities and generalization. For example, in medical image super-resolution, medical expertise and prior information can be incorporated to optimize the model's generative capabilities and generalization.

[0469] The apparatus of the present application embodiment will now be described with reference to Figures 10 and 11. It should be understood that the apparatus described below is capable of performing the methods of the foregoing embodiments of the present application. To avoid unnecessary repetition, repeated descriptions will be appropriately omitted when introducing the apparatus of the present application embodiment.

[0470] Figure 10 shows a schematic block diagram of an apparatus provided in an embodiment of this application. The apparatus 1800 shown in Figure 10 can be used to perform the method shown in Figure 6 or Figure 9.

[0471] As shown in Figure 10, the device 1800 may include: a first acquisition module 1810, a second acquisition module 1820, and a processing module 1830.

[0472] As one possible implementation, the device 1800 can be used to perform the method 900 of FIG9.

[0473] The first acquisition module 1810 is used to acquire the first image.

[0474] The second acquisition module 1820 is used to acquire the first text, which indicates the content in the second image to be generated.

[0475] Processing module 1830 is configured to: perform iterative denoising processing on an initial noisy representation at multiple time steps based on the first text and the first image to generate a second image, wherein the resolution of the second image is greater than the resolution of the first image, wherein during the denoising processing of the noisy representation at time step t in the multiple time steps...

[0476] The first attention operation is performed on the first text representation and the first noisy representation of the first text through the first attention layer in the image generation model to obtain the first operation result;

[0477] A second attention operation is performed on the first image representation and the second noisy representation of the first image through the second attention layer in the image generation model to obtain the second operation result. The fusion result of the first operation result and the second operation result is used to determine the denoising result at time step t. The denoising result at time step t is used to determine the second image. The first noisy representation and the second noisy representation are obtained by processing the noisy representation at time step t. The noisy representation at time step t is the initial noisy representation, or the noisy representation at time step t is obtained by denoising the initial noisy representation. t is a positive integer.

[0478] The first acquisition module 1810 and the second acquisition module 1820 can be the same module or different modules.

[0479] Optionally, the image generation model further includes a first text branch, a first image branch, and a first noise branch. The first text representation of the first text is obtained by processing the initial text representation of the first text through the first text branch. The first image representation of the first image is obtained by processing the initial image representation of the first image through the first image branch. The first noisy representation is obtained by processing the noisy representation at time step t through the first sub-branch of the first noise branch. The second noisy representation is obtained by processing the noisy representation at time step t through the second sub-branch of the first noise branch.

[0480] Optionally, the first text branch, the first image branch, and the first noise branch are constructed based on the transformation transformer layer.

[0481] Optionally, the image generation model includes a first diffusion block and a second diffusion block, which have the same structure. The first diffusion block includes a first text branch, a first noise branch, a first image branch, a first attention layer, and a second attention layer. During the denoising process of the noisy representation at time step t, the input of the second diffusion block includes the second text representation of the first text, the second image representation of the first image, and the third noisy representation. The second text representation of the first text is obtained by processing the result of the first operation through the first text branch; the second image representation of the first image is obtained by processing the result of the second operation through the first image branch; and the third noisy representation is obtained by processing the fusion result of the first operation result and the second operation result through the first noise branch.

[0482] Optionally, the first text is the descriptive text of the first image.

[0483] Optionally, the image generation model is obtained by adjusting the parameters of the initial image generation model based on the training dataset with the goal of reducing diffusion loss. The training dataset includes multiple image-text pairs, each consisting of a sample image, a target sample image, and sample text with matching relationships. The resolution of the sample image is smaller than that of the target sample image. The diffusion loss includes the difference between the predicted noise and the true noise at time step t', and / or the difference between the denoising result at time step t' and the true noisy representation at time step t'+1. Time step t' belongs to multiple time steps, and the true noisy representation at time step t' is obtained by adding noise to the image representation of the target sample image. t' is a positive integer, and t'+1 belongs to multiple time steps. The true noisy representation at time step t'+1 is obtained by adding noise to the image representation of the target sample image, or the true noisy representation at time step t'+1 is the image representation of the target sample image. The predicted noise and / or the denoising result at time step t' are obtained by processing the true noisy representation at time step t' using the initial image generation model based on the initial image representation of the sample image and the initial text representation of the sample text.

[0484] Optionally, the initial image generation model is built based on a pre-trained model. The image generation model is obtained by adjusting the parameters of the initial image generation model other than the parameters of the pre-trained model with the goal of reducing diffusion loss. The pre-trained model is used to perform the text-to-image task.

[0485] Optionally, the first text branch, the first sub-branch, and the first attention layer belong to the pre-trained model, while the first image branch, the second sub-branch, and the second attention layer do not belong to the pre-trained model.

[0486] Optionally, in the p-th iteration of the training process, the initial image representation of the sample image is generated based on the masking result, which is obtained by performing a masking operation on the sample image. The masking ratio used in the masking operation is negatively correlated with the value of p, where p is a positive integer.

[0487] For a detailed description, please refer to Method 900 above; it will not be repeated here.

[0488] As another possible implementation, device 1800 can be used to execute method 600 shown in FIG. 6. When used to execute method 600, the first acquisition module 1810 and the second acquisition module 1820 are the same acquisition module.

[0489] The first acquisition module 1810 is used to train the dataset. The training dataset includes multiple image-text pairs. Each image-text pair includes a sample image, a target sample image, and sample text that have a matching relationship. The resolution of the sample image is smaller than that of the target sample image.

[0490] Processing module 1830 is used for:

[0491] The initial image generation model is trained using the training dataset to obtain a trained image generation model. This model iteratively denoises the initial noisy representation across multiple time steps based on the input image and text, thereby generating the image.

[0492] The initial image generation model includes a first attention layer and a second attention layer. During training,

[0493] A first attention operation is performed on the first text representation and the fourth noisy representation of the sample text through a first attention layer to obtain a third operation result; a second attention operation is performed on the first image representation and the fifth noisy representation of the sample image through a second attention layer to obtain a fourth operation result; the predicted noise of time step t' in multiple time steps and / or the denoising result of time step t' are determined based on the fusion result of the third operation result and the fourth operation result. The predicted noise of time step t' and / or the denoising result of time step t' are used to adjust the parameters of the initial image generation model. The fourth noisy representation and the fifth noisy representation are obtained by processing the real noisy representation of time step t', where t' is a positive integer. The real noisy representation of time step t' is obtained by adding noise to the image representation of the target sample image.

[0494] Optionally, the processing module 1830 is specifically used to: adjust the parameters of the initial image generation model with the goal of reducing diffusion loss, wherein the diffusion loss includes the difference between the predicted noise at time step t' and the true noise at time step t', and / or the difference between the denoising result at time step t' and the true noisy representation at time step t'+1, wherein the true noisy representation at time step t'+1 is obtained by adding noise to the image representation of the target sample image, or the true noisy representation at time step t'+1 is the image representation of the target sample image.

[0495] Optionally, the initial image generation model includes a first text branch, a first image branch, and a first noise branch. The first text representation of the sample text is obtained by processing the initial text representation of the sample text through the first text branch. The first image representation of the sample image is obtained by processing the initial image representation of the sample image through the first image branch. The fourth noisy representation is obtained by processing the true noisy representation at time step t' through the first sub-branch of the first noise branch. The fifth noisy representation is obtained by processing the true noisy representation at time step t' through the second sub-branch of the first noise branch.

[0496] Optionally, the initial image generation model is built based on a pre-trained model used to perform the text-to-image task, and the processing module is specifically used to: adjust other parameters in the initial image generation model besides the parameters of the pre-trained model with the goal of reducing diffusion loss, and freeze the parameters of the pre-trained model.

[0497] Optionally, the first text branch, the first sub-branch, and the first attention layer belong to the pre-trained model, while the first image branch, the second sub-branch, and the second attention layer do not belong to the pre-trained model.

[0498] Optionally, the parameters of the first image branch are initialized based on the parameters of the first sub-branch.

[0499] Optionally, the parameters of the output layer of the second attention layer are initialized with zero.

[0500] Optionally, the processing module 1830 is further configured to: perform a masking operation on the sample image in the p-th iteration during the training process to obtain a masking result, where p is a positive integer; and generate an initial image representation of the sample image based on the masking result.

[0501] Optionally, the masking ratio used in the masking operation is negatively correlated with the value of p.

[0502] Optionally, when p is less than the first threshold, the mask ratio used in the masking operation is negatively correlated with the value of p, and / or when p is greater than or equal to the first threshold, the mask ratio used in the masking operation is the first ratio.

[0503] Optionally, the sample text is the descriptive text of the sample image, and / or the descriptive text of the target sample image.

[0504] Each module in device 1800 can be implemented in software or in hardware. For example, the implementation of processing module 1830 will be described below. Similarly, the implementation methods of other modules can be the same as those of processing module 1830.

[0505] As an example of a software functional unit, processing module 1830 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, processing module 1830 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0506] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0507] As an example of a hardware functional unit, the processing module 1830 may include at least one computing device, such as a server. Alternatively, the processing module 1830 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0508] The processing module 1830 includes multiple computing devices that can be distributed within the same region or in different regions. These computing devices can be distributed within the same Availability Zone (AZ), within the same Virtual Private Cloud (VPC), or across multiple VPCs. Furthermore, the multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0509] It should be noted that the division of units in the above device is only a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. In other embodiments, the processing module 1830 can be used to execute any step in the method described above, and other modules can be used to implement any step described above. The steps that each module is responsible for implementing can be specified as needed. By having each module implement different steps described above, all functions of the device 1800 can be achieved.

[0510] Figure 11 is a schematic block diagram of the apparatus provided in an embodiment of this application. The apparatus 1900 may include a processor 1910, a transceiver 1920, and a memory 1930. The processor 1910, transceiver 1920, and memory 1930 are connected via internal interconnection paths. The memory 1930 is used to store instructions, and the processor 1910 is used to execute the instructions stored in the memory 1930 to receive / send data via the transceiver 1920. Optionally, the memory 1930 may be coupled to the processor 1910 via an interface or integrated with the processor 1910.

[0511] It should be noted that the transceiver 1920 mentioned above may include, but is not limited to, transceiver devices such as input / output interfaces, to enable communication between device 1900 and other devices or communication networks.

[0512] The memory 1930 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM).

[0513] In one implementation, the processor 1910 can be a circuit with instruction read and execute capabilities, such as a central processing unit (CPU), microprocessor, or digital signal processor (DSP). In another implementation, the processor 1910 can implement certain functions through the logical relationships of hardware circuits. These logical relationships can be fixed or reconfigurable. For example, the processor 1910 can be a hardware circuit implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as a field-programmable gate array (FPGA). In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the processor loading instructions to implement some or all of the functions of the aforementioned units.

[0514] This application also provides an electronic device, which may include the above-described device 1800 or device 1900.

[0515] This application also provides a computing device cluster, which includes at least one computing device. The computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smartphone. The computing device cluster includes at least one device 1900.

[0516] In some possible implementations, the memory 1930 of one or more computing devices in the computing device cluster may also store partial instructions for executing the aforementioned method. In other words, a combination of one or more computing devices can jointly execute the instructions for executing the aforementioned method.

[0517] It should be noted that the memory 1930 in different computing devices within the computing device cluster can store different instructions, each used to execute a portion of the steps described above. That is, the instructions stored in the memory 1930 of different computing devices can implement the functions of one or more modules in device 1900.

[0518] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc.

[0519] This application also provides a computer program product, which includes computer program code that, when run on a computer, causes the computer to perform the methods described in the above embodiments.

[0520] This application also provides a computer-readable medium storing program code that, when run on a computer, causes the computer to perform the methods described in the above embodiments.

[0521] This application also provides a chip, including circuitry, for performing the methods described in the above embodiments.

[0522] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules within the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, power-on erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0523] It should also be understood that, in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0524] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0525] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0526] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0527] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0528] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0529] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be covered.

Claims

1. A method of generating an image, characterized by, include: Get the first image; Obtain the first text, which indicates the content in the second image to be generated; Based on the first text and the first image, an initial noisy representation is iteratively denoised at multiple time steps to generate a second image, wherein the resolution of the second image is greater than that of the first image. During the denoising process of the noisy representation of time step t among the multiple time steps: A first attention operation is performed on the first text representation and the first noisy representation of the first text through the first attention layer in the image generation model to obtain the first operation result; A second attention operation is performed on the first image representation and the second noisy representation of the first image through the second attention layer in the image generation model to obtain a second operation result. The fusion result of the first operation result and the second operation result is used to determine the denoising result at time step t. The denoising result at time step t is used to determine the second image. The first noisy representation and the second noisy representation are obtained by processing the noisy representation at time step t. The noisy representation at time step t is the initial noisy representation, or the noisy representation at time step t is obtained by denoising the initial noisy representation. t is a positive integer.

2. The method of claim 1, wherein, The image generation model further includes a first text branch, a first image branch, and a first noise branch. The first text representation of the first text is obtained by processing the initial text representation of the first text through the first text branch. The first image representation of the first image is obtained by processing the initial image representation of the first image through the first image branch. The first noisy representation is obtained by processing the noisy representation of time step t through the first sub-branch of the first noise branch. The second noisy representation is obtained by processing the noisy representation of time step t through the second sub-branch of the first noise branch.

3. The method of claim 2, wherein, The first text branch, the first image branch, and the first noise branch are constructed based on the transformer layer.

4. The method according to claim 2 or 3, characterized in that, The image generation model includes a first diffusion block and a second diffusion block, which have the same structure. The first diffusion block includes a first text branch, a first noise branch, a first image branch, a first attention layer, and a second attention layer. During the denoising process of the noisy representation at time step t, the input of the second diffusion block includes the second text representation of the first text, the second image representation of the first image, and the third noisy representation. The second text representation of the first text is obtained by processing the result of the first operation through the first text branch; The second image representation of the first image is obtained by processing the result of the second operation through the first image branch; The third noisy characterization is obtained by processing the fusion result of the first operation result and the second operation result through the first noise branch.

5. The method according to any one of claims 1 to 4, characterized in that, The first text is the descriptive text of the first image.

6. The method according to any one of claims 2 to 4, characterized in that, The image generation model is obtained by adjusting the parameters of an initial image generation model based on a training dataset with the goal of reducing diffusion loss. The training dataset includes multiple image-text pairs, each consisting of a matching sample image, a target sample image, and sample text. The resolution of the sample image is smaller than that of the target sample image. The diffusion loss includes the difference between the predicted noise at time step t' and the actual noise at time step t', and / or the difference between the denoising result at time step t' and the actual noisy representation at time step t'+1. Time step t' belongs to the multiple time steps, and the actual noisy representation at time step t'... The representation is obtained by adding noise to the image representation of the target sample image, where t' is a positive integer, and t'+1 belongs to the plurality of time steps. The true noisy representation of time step t'+1 is obtained by adding noise to the image representation of the target sample image, or the true noisy representation of time step t'+1 is the image representation of the target sample image. The predicted noise of time step t' and / or the denoising result of time step t' is obtained by the initial image generation model processing the true noisy representation of time step t' based on the initial image representation of the sample image and the initial text representation of the sample text.

7. The method of claim 6, wherein, The initial image generation model is built based on a pre-trained model. The image generation model is obtained by adjusting other parameters in the initial image generation model, except for the parameters of the pre-trained model, with the goal of reducing the diffusion loss. The pre-trained model is used to perform the text-to-image task.

8. The method of claim 7, wherein, The first text branch, the first sub-branch, and the first attention layer belong to the pre-trained model, while the first image branch, the second sub-branch, and the second attention layer do not belong to the pre-trained model.

9. The method of claim 8, wherein, In the p-th iteration of the training process, the initial image representation of the sample image is generated based on the masking result, which is obtained by performing a masking operation on the sample image. The masking ratio used in the masking operation is negatively correlated with the value of p, where p is a positive integer.

10. A training method for an image generation model, characterized in that, include: Obtain a training dataset, which includes multiple image-text pairs. Each image-text pair includes a sample image, a target sample image, and sample text that have a matching relationship. The resolution of the sample image is smaller than that of the target sample image. The initial image generation model is trained using the training dataset to obtain a trained image generation model. The image generation model iteratively denoises the initial noisy representation at multiple time steps based on the input image and text to the model, thereby generating the image. The initial image generation model includes a first attention layer and a second attention layer. During training... A first attention operation is performed on the first text representation and the fourth noisy representation of the sample text through the first attention layer to obtain a third operation result; a second attention operation is performed on the first image representation and the fifth noisy representation of the sample image through the second attention layer to obtain a fourth operation result; the predicted noise of time step t' in the plurality of time steps is determined according to the fusion result of the third operation result and the fourth operation result, and / or the denoising result of time step t', the predicted noise of time step t' and / or the denoising result of time step t' are used to adjust the parameters of the initial image generation model, the fourth noisy representation and the fifth noisy representation are obtained by processing the real noisy representation of time step t', where t' is a positive integer, and the real noisy representation of time step t' is obtained by adding noise to the image representation of the target sample image.

11. The training method of claim 10, wherein, The step of training the initial image generation model based on the training dataset includes: The parameters of the initial image generation model are adjusted with the goal of reducing diffusion loss. The diffusion loss includes the difference between the predicted noise at time step t' and the actual noise at time step t', and / or the difference between the denoising result at time step t' and the actual noisy representation at time step t'+1. The actual noisy representation at time step t'+1 is obtained by adding noise to the image representation of the target sample image, or the actual noisy representation at time step t'+1 is the image representation of the target sample image.

12. The training method according to claim 10 or 11, characterized in that, The initial image generation model includes a first text branch, a first image branch, and a first noise branch. The first text representation of the sample text is obtained by processing the initial text representation of the sample text through the first text branch. The first image representation of the sample image is obtained by processing the initial image representation of the sample image through the first image branch. The fourth noisy representation is obtained by processing the true noisy representation of time step t' through the first sub-branch of the first noise branch. The fifth noisy representation is obtained by processing the true noisy representation of time step t' through the second sub-branch of the first noise branch.

13. The training method of claim 12, wherein, The initial image generation model is built based on a pre-trained model used to perform text-to-image tasks, and the parameters of the initial image generation model are adjusted to reduce diffusion loss, including: The parameters of the initial image generation model, excluding those of the pre-trained model, are adjusted with the goal of reducing diffusion loss, and the parameters of the pre-trained model are frozen.

14. The training method of claim 13, wherein, The first text branch, the first sub-branch, and the first attention layer belong to the pre-trained model, while the first image branch, the second sub-branch, and the second attention layer do not belong to the pre-trained model.

15. The training method according to claim 13 or 14, characterized in that, The parameters of the first image branch are initialized based on the parameters of the first sub-branch.

16. The training method according to claim 14 or 15, characterized in that, The parameters of the output layer of the second attention layer are initialized with zero.

17. The training method of any one of claims 12 to 16, wherein, The training method also includes: In the p-th iteration of the training process, a masking operation is performed on the sample image to obtain a masking result, where p is a positive integer; An initial image representation of the sample image is generated based on the masking result.

18. The training method of claim 17, wherein, The masking ratio used in the masking operation is negatively correlated with the value of p.

19. The training method of claim 18, wherein, When p is less than the first threshold, the masking ratio used in the masking operation is negatively correlated with the value of p, and / or, When p is greater than or equal to the first threshold, the masking ratio used in the masking operation is the first ratio.

20. The training method of any one of claims 10-19, wherein, The sample text is the descriptive text of the sample image, and / or the descriptive text of the target sample image.

21. An apparatus for image generation, characterized by include: The first acquisition module is used to acquire the first image; The second acquisition module is used to acquire the first text, which indicates the content in the second image to be generated. Processing module, used for: Based on the first text and the first image, an initial noisy representation is iteratively denoised at multiple time steps to generate a second image, the resolution of the second image being greater than that of the first image. Specifically, during the denoising process of the noisy representation at time step t, among the multiple time steps... A first attention operation is performed on the first text representation and the first noisy representation of the first text through the first attention layer in the image generation model to obtain the first operation result; A second attention operation is performed on the first image representation and the second noisy representation of the first image through the second attention layer in the image generation model to obtain a second operation result. The fusion result of the first operation result and the second operation result is used to determine the denoising result at time step t. The denoising result at time step t is used to determine the second image. The first noisy representation and the second noisy representation are obtained by processing the noisy representation at time step t. The noisy representation at time step t is the initial noisy representation, or the noisy representation at time step t is obtained by denoising the initial noisy representation. t is a positive integer.

22. The apparatus of claim 21, wherein, The image generation model further includes a first text branch, a first image branch, and a first noise branch. The first text representation of the first text is obtained by processing the initial text representation of the first text through the first text branch. The first image representation of the first image is obtained by processing the initial image representation of the first image through the first image branch. The first noisy representation is obtained by processing the noisy representation of time step t through the first sub-branch of the first noise branch. The second noisy representation is obtained by processing the noisy representation of time step t through the second sub-branch of the first noise branch.

23. The apparatus of claim 22, wherein, The image generation model includes a first diffusion block and a second diffusion block, which have the same structure. The first diffusion block includes a first text branch, a first noise branch, a first image branch, a first attention layer, and a second attention layer. During the denoising process of the noisy representation at time step t, the input of the second diffusion block includes the second text representation of the first text, the second image representation of the first image, and the third noisy representation. The second text representation of the first text is obtained by processing the result of the first operation through the first text branch; The second image representation of the first image is obtained by processing the result of the second operation through the first image branch; The third noisy characterization is obtained by processing the fusion result of the first operation result and the second operation result through the first noise branch.

24. An apparatus for training an image generation model, comprising: include: The acquisition module is used to acquire a training dataset, which includes multiple image-text pairs. Each image-text pair includes a sample image, a target sample image, and sample text that have a matching relationship. The resolution of the sample image is smaller than the resolution of the target sample image. The processing module is used to train the initial image generation model based on the training dataset to obtain a trained image generation model. The image generation model is used to iteratively denoise the initial noisy representation at multiple time steps based on the image and text input to the image generation model, to obtain the generated image. The initial image generation model includes a first attention layer and a second attention layer. During training... A first attention operation is performed on the first text representation and the fourth noisy representation of the sample text through the first attention layer to obtain a third operation result; a second attention operation is performed on the first image representation and the fifth noisy representation of the sample image through the second attention layer to obtain a fourth operation result; the predicted noise of time step t' in the plurality of time steps is determined according to the fusion result of the third operation result and the fourth operation result, and / or the denoising result of time step t', the predicted noise of time step t' and / or the denoising result of time step t' are used to adjust the parameters of the initial image generation model, the fourth noisy representation and the fifth noisy representation are obtained by processing the real noisy representation of time step t', where t' is a positive integer, and the real noisy representation of time step t' is obtained by adding noise to the image representation of the target sample image.

25. The exercise device of claim 24, wherein, The initial image generation model includes a first text branch, a first image branch, and a first noise branch. The first text representation of the sample text is obtained by processing the initial text representation of the sample text through the first text branch. The first image representation of the sample image is obtained by processing the initial image representation of the sample image through the first image branch. The fourth noisy representation is obtained by processing the true noisy representation of time step t' through the first sub-branch of the first noise branch. The fifth noisy representation is obtained by processing the true noisy representation of time step t' through the second sub-branch of the first noise branch.

26. The exercise device of claim 25, wherein, The initial image generation model is built based on a pre-trained model, which is used to perform text-to-image tasks, and the processing module is specifically used for: The parameters of the initial image generation model, excluding those of the pre-trained model, are adjusted with the goal of reducing diffusion loss, and the parameters of the pre-trained model are frozen.

27. The training device of claim 25 or 26, wherein, The processing module is also used for: In the p-th iteration of the training process, a masking operation is performed on the sample image to obtain a masking result, where p is a positive integer; An initial image representation of the sample image is generated based on the masking result.

28. The exercise device of claim 27, wherein, The masking ratio used in the masking operation is negatively correlated with the value of p.

29. A computing device, comprising: It includes a processor and a memory, the processor being configured to execute instructions stored in the memory to cause the computing device to perform the method as described in any one of claims 1 to 9, or to perform the method as described in any one of claims 10 to 20.

30. A computer program product comprising instructions, wherein: When the instructions are executed by the computing device, the computing device performs the method as described in any one of claims 1 to 9, or performs the method as described in any one of claims 10 to 20.

31. A computer readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computing device, enable the computing device to perform the method as described in any one of claims 1 to 9, or to perform the method as described in any one of claims 10 to 20.