Image generation based on region adaptive sampling

Abstract

According to implementations of the disclosure, a solution for image generation is provided. According to the solution, in a plurality of iterative rounds of a machine learning model, an input image for a current round is obtained; a plurality of image blocks in the input image are classified into a first category and a second category to obtain at least one image block of the first category and at least one image block of the second category; a first noise corresponding to the at least one image block of the first category is determined; a second noise for the at least one image block of the second category in the previous round is obtained; and the first noise and the second noise are removed respectively from the plurality of image blocks, to obtain an input image of a next round; and a target image is determined.

Description

IMAGE GENERATION BASED ON REGION ADAPTIVE SAMPLINGBACKGROUND

[0001] Diffusion Models, also referred to as diffusion probability models, are a type of generative models that generate data by simulating diffusion processes. Diffusion models have been proven to be efficient probabilistic generative models, producing high-quality data across various domains. Applications of the diffusion models include image synthesis, image editing, video generation, and the like.SUMMARY

[0002] According to implementations of the present disclosure, a solution for image generation based on region adaptive sampling is provided. In this solution, in a plurality of iterative rounds of a machine learning model, an input image for a current round is obtained, where the machine learning model is configured to generate a target image by iteratively performing denoising on input information; a plurality of image blocks in the input image are classified into a first category and a second category at least based on respective noises corresponding to the plurality of image blocks in the input image in a previous round, to obtain at least one image block of the first category and at least one image block of the second category; a first noise corresponding to the at least one image block of the first category is determined by the machine learning model in the current round; a second noise determined by the machine learning model for the at least one image block of the second category' in the previous round is obtained; and the first noise and the second noise are removed respectively from the plurality' of image blocks of the input image, to obtain an input image of a next round; and the target image generated by the machine learning model after the plurality of iterative rounds is determined. According to the embodiments of the present disclosure, the plurality' of image blocks in the input image may be classified into different categories based on a noise of a previous round. For the image block of the first category', the noise of the current round may be determined by the machine learning model. For the image block of the second category, the noise of the previous round may be used. In this way, sampling ratios of different regions in the input image may be dynamically adjusted, and the efficiency of image generation is improved.

[0003] This section is provided to introduce the selection of objects in a simplified form, which will be further described below in the Detailed Description. This section is not intended to identify key features or primary' features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.BRIEF DESCRIPTION OF DRAWINGS

[0004] FIG. 1 illustrates a block diagram of an example environment in which a plurality ofimplementations of the present disclosure can be implemented;

[0005] FIG. 2 illustrates a schematic diagram of similarity of output noises for a region of interest of a diffusion model in adjacent sampling steps according to some implementations of the present disclosure;

[0006] FIG. 3 illustrates a schematic diagram of an example architecture for image generation based on region adaptive sampling according to some implementations of the present disclosure;

[0007] FIG. 4 illustrates a schematic diagram of a process of applying a self-attention mechanism to at least one image block of a first category according to some implementations of the present disclosure;

[0008] FIG. 5 illustrates a flowchart of a process of operations performed at a plurality of iterative rounds of a machine learning model according to some implementations of the present disclosure;

[0009] FIG. 6 shows a flowchart of a process of image generation based on region adaptive sampling according to some implementations of the present disclosure; and

[0010] FIG. 7 shows a schematic block diagram of an electronic device capable of implementing a plurality of implementations of the present disclosure.DETAILED DESCRIPTION

[0011] The present disclosure will now be discussed with reference to several example implementations. It should be understood that these implementations are discussed merely to enable those of ordinary skill in the art to better understand and thus implement the present disclosure, rather than implying any limitation on the scope of the present disclosure.

[0012] As used herein, the term “comprising / including” and variations thereof are to be interpreted as open-ended terms that mean "‘comprising / including, but not limited to”. The term “based on” is to be interpreted as “based at least in part on”. The terms “an implementation” and “one implementation” are to be interpreted as “at least one implementation”. The term “another implementation” is to be interpreted as “at least one other implementation”. The terms “first,” “second,” and the like may refer to different or identical objects. Other explicit and implicit definitions may also be included below.

[0013] It is noted that the headings of any of the sections / subsections provided herein are not limiting. Various implementations are described herein throughout, and any type of implementation may be included under any section / subsection. Further, implementations described in any section / subsection may be combined in any manner with any other implementations described in the same section / subsection and / or different sections / sections.

[0014] Herein, unless explicitly stated, performing a step “in response to A” does not mean that the step is performed immediately after “A”, but may include one or more intermediate steps.

[0015] As used herein, the term “model” may learn associations between respective inputs andoutputs from training data so that corresponding outputs may be generated for a given input after training is completed. The generation of the model may be based on machine learning techniques. Deep learning (DL) is a machine learning algorithm that processes inputs and provides corresponding outputs by using a multi-layer processing unit. A neural network model is one example of a deep learning-based model. A ‘'model’7may also be referred to herein as a “machine learning model,” a “learning model,” a “machine learning network,” or a “learning network,” these terms are used interchangeably herein.

[0016] The diffusion process involved in the diffusion models is inspired by physical processes such as thermal diffusion. The diffusion models include a forward diffusion process and a reverse diffusion process. The diffusion models generate new data samples by simulating a forward diffusion process that adds noises in steps and then learning how to reverse this process.

[0017] During the forward diffusion process, noises are gradually added to the data, making the data increasingly random through a series of steps until the data resembles a pure noise. This process may be regarded as a Markov chain, and at each step, a Gaussian noise is added to the data. The forward diffusion process may be expressed as:(1 — at)Z), where xtis noise data at step t, and atis used to control the amount of noise added. The forward diffusion process is performed in model training, and the data for adding noise is a training sample.

[0018] During the reverse diffusion process (or referred to as a reverse denoising process), the model learns how to reverse the step of adding noise. From the pure noise, the diffusion model gradually removes noise to generate data matching the training distribution. The reverse diffusion process is typically simulated using a neural network that predicts the noise added at each step: Pd(xt-i\xt=^Cxt-i>uext> -ad(xti 0) , where ueand aeare model parameters by learning. After model training is completed, the model performing the reverse diffusion process may first sample from the noise distribution and iteratively denoise until the expected data is obtained.

[0019] In diffusion models, a time step refers to the number of steps of adding noise during forward diffusion. The total number of steps T is usually a predetermined value, indicating how many steps are required in the process of converting the original data into the pure noise. At each time step t, Gaussian noise is added to the data according to a predetermined noise scheme. This process is continuous, and each step depends on the result of the previous step.

[0020] When generating data, inference steps (or referred to as sampling steps) of a diffusion model refer to the number of steps required for restoring the pure noise to original data in a reverse diffusion process. The number of inference steps directly affects the quality and speed of generating data. Generally, the more the inference steps are, the higher the quality of the generateddata is, but the computational cost and time are also increased. In practical applications, the generation quality and efficiency may be balanced by adjusting the number of inference steps. In some embodiments, the inference step corresponds to the time step, and each inference step may correspond to one or more time steps. For example, if the total time steps of a diffusion model are 1000 steps and the inference steps are set to 50 steps, then each inference step may correspond to 20 time steps.

[0021] FIG. 1 illustrates a schematic diagram of an example environment 100 in which implementations of the present disclosure can be implemented. As shown in FIG. 1, the environment 100 includes an electronic device 110. A machine learning model 140 may be deployed in the electronic device 110 for processing an input image 120 to obtain an output image 130.

[0022] In some embodiments, the machine learning model 140 may be constructed based on a diffusion model. The machine learning model 140 may predict a noise of the input image 120 in a current round of a plurality of iterative rounds and denoise the input image 120 to obtain the output image 130. The output image 130 may be used as an input image of a next round for the machine learning model 140 to continue denoising.

[0023] In some embodiments, the diffusion model utilizes a convolutional neural network (for example. U-Net) to predict a noise of the input image 120.

[0024] In some embodiments, the diffusion model utilizes a transformer to predict a noise of the input image 120.

[0025] In FIG. 1 , the electronic device 110 may be any sy stem having a computing capability, for example, various computing devices / systems, terminal devices, servers, and the like. The terminal device may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, or any combination of the foregoing, including accessories and peripherals of these devices, or any combination thereof. Servers include, but are not limited to, mainframes, edge computing nodes, computing devices in a cloud environment, and the like.

[0026] It should be understood that the components and arrangements in the environment shown in FIG. 1 are merely examples, and a computing system suitable for implementing implementations described in the present disclosure may include one or more different components, other components, and / or different arrangements.

[0027] It should be understood that the structure and function of various elements in the environment 100 are described for an example purpose only, and do not imply any limitation on the scope of the present disclosure.

[0028] As mentioned above, diffusion models have been proven to generate high quality data in various domains. However, generating a sample (for example, an image) using a diffusion model involves solving a generative stochastic differential equation (SDE) or an ordinary differential equation (ODE) in inverse time, which requires multiple times of sequential forward propagation through a neural network. This sequential processing limits the real-time performance of the diffusion model and has a large overhead on computing resources. To address the problem of high inference cost in diffusion models, various acceleration techniques have been proposed from different perspectives. Some approaches reduce the number of sampling steps to accelerate the sampling process in the diffusion models. These approaches require additional training, such as progressive distillation, consistency models, and rectified flow. For example, rectified flow learns the ODE to follow straight paths between the standard normal distribution and the distribution of the training dataset. These straight paths significantly reduce the distance between the two distributions, which in turn lower the number of sampling steps needed.

[0029] There are also training-free methods to either reduce the number of sampling steps or decrease the computational burden within each step. For instance, a diffusion probabilistic model (DPM)-solver introduces a formulation that enhances the solution process of diffusion DDEs. The U-Net architecture is leveraged to cache and retrieve features across adjacent stages, allowing for the skipping of certain downsampling and upsampling operations during the diffusion process. However, these approaches treat all regions in the image uniformly, ignoring the varying complexity across different parts of the image. This uniform treatment can lead to significant computational inefficiency, as not all regions require the same level of processing.

[0030] To resolve the foregoing problems, implementations of the present disclosure provide a solution for image generation based on region adaptive sampling. According to implementations of the present disclosure, in this solution, in a plurality of iterative rounds of a machine learning model, an input image for a current round is obtained, where the machine learning model is configured to generate a target image by iteratively performing denoising on input information; a plurality of image blocks in the input image are classified into a first category and a second category at least based on respective noises corresponding to the plurality of image blocks in the input image in a previous round, to obtain at least one image block of the first category and at least one image block of the second category; a first noise corresponding to the at least one image block of the first category is determined by the machine learning model in the current round; a second noise determined by the machine learning model for the at least one image block of the second category in the previous round is obtained; and the first noise and the second noise are removed respectively from the plurality of image blocks of the input image, to obtain an input image of a next round; and the target image generated by the machine learning model after theplurality of iterative rounds is determined. According to the embodiments of the present disclosure, the plurality of image blocks in the input image may be classified into different categories based on a noise of a previous round. For the image block of the first category, the noise of the current round may be determined by the machine learning model. For the image block of the second category, the noise of the previous round may be used. In this way, sampling ratios of different regions in the input image may be dynamically adjusted, and the efficiency of image generation is improved.

[0031] To better describe the embodiments of the present disclosure, the inspirations behind the solution of image generation based on region-adaptive sampling, which is proposed in the present disclosure, will be first introduced below.

[0032] With the development of diffusion models, embodiments of the present disclosure propose performing different sampling steps on different regions in an image. Some previous diffusion models use U-Net as a backbone network, and due to the limitation of the structure of U-Net, it is necessary to maintain the original spatial resolution of the input samples to support the convolution operation. The U-Net structure needs to perform unified processing on all image regions due to fixed input. However, with the evolution of the diffusion model structure, a current model architecture, for example, a model structure based on a diffusion transformer (DiT), has become an architecture capable of accommodating flexible token inputs. Such a diffusion model allows flexible processing of any number of tokens, which enables the possibility of allocating different sampling steps to different regions in an image.

[0033] To further explore the feasibility of the solution proposed by the present disclosure, the present disclosure visualizes several diffusion process outputs at varying sampling steps. Two key phenomena have been observed. The first phenomenon is that the regions of focus in adjacent steps exhibit considerable continuity during the later stages of diffusion, and the second phenomenon is that in each sampling step, the diffusion model tends to focus on specific semantically meaningful areas within the image. This pattern is similar to the process of selectively refining certain areas. This observation suggests that in each adjacent step the brushes used by the diffusion model are similar, and hence, the areas refined by the diffusion model remain consistent. Thus, areas that the model momentarily ignores could potentially be excluded from the computationally intensive diffusion processing, allowing the model to focus more on regions of interest.

[0034] FIG. 2 shows a schematic diagram 200 of similarity of output noises for a region of interest of a diffusion model in adjacent sampling steps according to some implementations of the present disclosure. As show n in FIG. 2, the similarity of the output noises for the region of interest of the diffusion model in adjacent sampling steps is computed by using a normalized discountedcumulative gain (NDCG). The NDCG between 0.95 and 1 indicates that the region of interest of the diffusion model has a high degree of continuity in adjacent sampling steps. This continuity motivated the present disclosure to propose a solution for image generation based on region-adaptive sampling, which allocates different sampling ratios (sometimes referred to as sampling steps) to different regions according to the continuity of output noises for different regions.

[0035] Some example implementations of the present disclosure will be described in more detail below with reference to the accompanying drawings.

[0036] FIG. 3 illustrates a schematic diagram of an example architecture 300 for image generation based on region adaptive sampling according to some implementations of the present disclosure. As shown in FIG. 3, the architecture 300 includes the machine learning model 140 configured to generate the target image by iteratively performing denoising on the input information in the plurality of iterative rounds as previously described. FIG. 3 only shows that the machine learning model 140 performs denoising on the input information in one round (for example, the current round t) of the plurality of iterative rounds.

[0037] The machine learning model 140 may be any machine learning model capable of predicting noise on the input information. In some implementations, the machine learning model 140 may be constructed based on a diffusion model. For example, the diffusion model may employ a transformer as a backbone network. Depending on the structure of the transformer, positional information of the input information (for example, an image) may be position embedded using techniques such as rotary position embedding (RoPE), therefore elements in the input information may be masked and reordered. In this way, a portion of the input information may be selected, so that the portion is processed by the machine learning model 140, thereby implementing flexible processing of the input information.

[0038] In the denoising process of the current round, an input image 305 (represented as Xt) for the current round is first obtained. In the inference process of the model, the input image 305 has a noise, and denoising needs to be performed on the input image 305.

[0039] In some implementations, the input image 305 may include a feature map in a latent space or an image in a pixel space. The image of the pixel space is an original representation of the image, and the feature map of the latent space captures intrinsic structures and features of the image of the pixel space. In this way, the machine learning model 140 may denoise images in different spaces, thereby improving application diversity of the solution provided by the present disclosure.

[0040] Different regions of the input image 305 typically have different complexities. For example, a foreground region (sometimes also referred to as a main object) of the input image 305 has a higher complexity, and a background region of the input image 305 has a lower complexity.Therefore, different regions of the input image 305 may be processed differently using the machine learning model 140.

[0041] To differentiate between different regions of the input image 305, a plurality of image blocks in the input image 305 are classified into a first category and a second category at least based on respective noises corresponding to the plurality of image blocks in the input image 305 in a previous round, to obtain at least one image block 320 of the first category (also referred to as a fast-update region) and at least one image block 325 of the second category (also referred to as a slow-update region). In some examples, the input image 305 is segmented into a plurality of image blocks for ease of processing. For example, if the size of the input image 305 is 16 x 16, and the size of the image block is 4 4, then the input image 305 may be divided into 1 image blocks. An image block may include a plurality of image units (for example, tokens). For example, if the size of an image block is 4 x 4, then the image block includes 16 image units.

[0042] In some implementations, at least one image block 320 of the first category may be associated with a main object in the input image 305, and the at least one image block 325 of the second category may be associated with a background in the input image. The main object in the input image 305 may indicate an important part in the input image 305, which is an object of primary’ interest to the viewer, for example, a person, an animal, an object, and the like. The background in the input image 305 may indicate an element in the input image 305 for setting off the main object, for example, the natural environment, the street, the sky, and the like.

[0043] In some implementations, respective metrics for the plurality of image blocks may be determined at least based on respective noises corresponding to the plurality of image blocks of the input image 305 in the previous round. The metric for an image block may indicate a level of denoising demand on the image block. Then, the plurality of image blocks in the input image may be classified into the first category and the second category’ based on the respective metrics for the plurality of image blocks. The level of denoising demand on the image block indicates the urgency of the image block for denoising or the importance of the image block in denoising. A higher level of denoising demand in the current round means that denoising of the image block can better improve image generation quality, and therefore denoising of the image block is worth and more important. For example, in general, the level of denoising demand on the background portion of an image is lower than the level of denoising demand on the main object or the foreground portion of the image.

[0044] The noise may be removed from the image block according to a noise corresponding to the image block in a previous round. Therefore, the noise may be considered as an extent to which the corresponding image block needs to be changed. The greater the noise is, the higher probability the corresponding image block is to be changed. In some examples, the metric for the image blockmay be determined based on the noise of the image block. For example, an absolute value of the noise of the image block may be determined as the metric for the image block. The larger the metric is, the higher the extent of denoising demand on the corresponding image block is. The plurality of image blocks are classified into the first category (that is, a category with a higher level of denoising demand) and the second category (that is, a category with a lower level of denoising demand) based on the metrics for the plurality of image blocks. In this way, the plurality of image blocks in the input image 305 may be accurately classified into different categories based on the levels of denoising demands on the image blocks.

[0045] In some implementations, the respective metrics for the plurality of image blocks may be sorted in a descending order. At least one image block corresponding to at least one metric sorted prior to a specified position are classified into the first category, and at least one image block corresponding to at least one metric sorted subsequent to the specified position is classified into the second category. The greater the metric for an image block, the higher the level of denoising demand on the image block. Therefore, the respective metrics for the plurality of image blocks may be sorted in a descending order, and at least one image block corresponding to top-ranked metrics is classified into the first category7. In some examples, the number of image blocks of the first category may be predetermined (for example, 5), then image blocks corresponding to metrics prior to the fifth metric (as an example of the specified position) may be classified into the first category, and image blocks corresponding to metrics subsequent to the fifth metric may be classified into the second category7.

[0046] In some implementations, the metrics for the plurality of image blocks may be compared with a predetermined threshold such that the plurality of image blocks are classified into the first category and the second category7. For example, at least one image block with the metric(s) greater than a predetermined threshold may be classified into the first category, and at least one image block w ith the metric(s) lower than or equal to the predetermined threshold may be classified into the second category. Certainly, the size comparison here depends on the definition of the metric. If the metric is defined such that a smaller metric indicates a greater level of denoising demand, then at least one image block with the metric lower than or equal to the predetermined threshold may be classified into the first category.

[0047] In some implementations, the plurality of image blocks in the input image 305 may be classified into the first category and the second category7based on a number of times a respective image block among the plurality of image blocks is classified into the second category in the historical rounds, in addition to the respective noises corresponding to the plurality7of image blocks of the input image 305 in the previous round. During the diffusion process, the image block of the first category generally requires more processing (for example, more sampling steps) thanthe image block of the second category. However, to continuously classify an image block into the second category so the processing of the image block is continuously reduced, it may lead to over-blurring or excessive noise in the final generated image. Therefore, when classifying the plurality of image blocks in the input image 305 into the first category and the second category, the number of times a respective image block among the plurality of image blocks is classified into the second category in the historical rounds also needs to be considered. For example, if an image block has been classified into the second category7more frequently in historical rounds, a probability of classifying this image block into the first category in the current round should be increased. In this way, the regions with relatively less importance in the input image 305 may also be adequately processed, thereby reducing the probability of over-blurring or excessive noise in the final generated image.

[0048] In some implementations, the at least one metric for the plurality of image blocks in the input image 305 may be determined based on a standard deviation of respective noises corresponding to the plurality of image blocks in the input image 305 in the previous round and the number of times a respective image block among the plurality of image blocks is classified into the second category7in historical rounds. Then, the plurality7of image blocks in the input image 305 are classified into the first category and the second category according to the metric. Through the observation of the present disclosure, the standard deviation of the noise may be used to distinguish different regions in the input image 305, and the standard deviation of the noise of the main object in the input image 305 may be lower than the standard deviation of the noise of the background. The metric determined based on the standard deviation of the respective noise corresponding to the plurality of image blocks in the previous round and the number of times a respective image block among the plurality of image blocks is classified into the second category in historical rounds may be represented as follows:where Rtrepresents a metric for a specific image block in a current round, mennpatcft(std(Ait)) represents an average of standard deviations of noises of all image units in the specific image block, Dpatchrepresents a number of times the specific image block is divided into the second category in historical rounds, k represents a scale factor, and exp () represents an exponential operation based on a natural constant e. In some examples, Rtmay be sorted in a descending order; at least one image block corresponding to Rtsorted prior to the specified position is classified into the first category; and at least one image block corresponding to Rtsorted subsequent to the specified position is classified into the second category7. It should be noted that Eq. (1) merely shows an example metric for classify ing the plurality7of image blocks of the input image 305 into the first category and the second category, and the metric may also be determinedbased on other factors (for example, an absolute value of noise), which is not limited in the present disclosure.

[0049] After the plurality of image blocks of the input image 305 are classified into the first category and the second category the at least one image block 320 of the first category and the at least one image block 325 of the second category are processed differently.

[0050] For the at least one image block 320 of the first category the machine learning model 140 determines a first noise 330 corresponding to the at least one image block 320 of the first category' in the current round. The at least one image block 320 of the first category corresponds to a relatively important region in the input image 305. Therefore, more refined processing is required, that is, the first noise 330 is determined by the machine learning model 140 in the cunent round. For example, the size of the input image 305 is 256 x 256, the size of the at least one image block 320 of the first category is 128 X 128, and the at least one image block with this size may be input into the machine learning model 140 to obtain the first noise 330. In some examples, the at least one image block 320 of the first category’ may be segmented into smaller image blocks and input into the machine learning model 140 to predict noise. In the case that the size of the at least one image block 320 of the first category is 128 X 128, it may be divided into 64 16 x 16 image blocks, which are then input into the machine learning model 140.

[0051] In some implementations, the first noise 330 is determined using the machine learning model 140 based on a round count of the current round, the at least one image block 320 of the first category; and prompt information for generating the target image. The prompt information may guide the machine learning model 140 to develop an image obtained by removing noise toward conforming to the prompt information. In an example, the round count of the current round is 5, the prompt information is “please generate a car”, then the machine learning model 140 may generate the first noise 330 to be removed in the fifth round, and the image obtained by removing the first noise 330 from the at least one image block 320 of the first class develops toward the direction of "a car”.

[0052] For the at least one image block 325 of the second category; a second noise 335 determined by the machine learning model 140 for the at least one image block 325 of the second category7in the previous round is obtained. The at least one image block 325 of the second category7corresponds to a relatively unimportant region in the input image 305, and the second noise 335 determined in the previous round may be used. In some examples, in each round, the image block of the first category7and the image block of the second category in the next round may be determined using a noise of respective image blocks in the round, and then the noise of the image block of the second category may be stored. Therefore, the noise of the image block of the second category stored in the previous round may be obtained in the current round. In this way, for the atleast one image block 325 of the second category, noise of the previous round may be used to reduce a computational amount and improve the efficiency of image generation.

[0053] After the first noise 330 and the second noise 335 are obtained, the first noise 330 and the second noise 335 are removed respectively from the plurality of image blocks of the input image 305, to obtain an input image 345 (represented byof a next round. Implementations of the present disclosure enable the at least one image block 320 of the first category to move in a new denoising direction determined in the current round, while the at least one image block 325 of the second category retains the denoising direction of the previous round. In some implementations, the first noise 330 and the second noise 335 may be combined to obtain a merged noise 340 of the current round. Removing the merged noise 340 from the plurality of image blocks of the input image 305 results in the input image 345 of the next round.

[0054] In some implementations, the first noise 330 corresponding to the at least one image block 320 of the first category may be determined based on a self-attention mechanism. Specifically, the self-attention mechanism may be applied to the at least one image block 320 of the first category, to obtain a feature representation of the at least one image block 320 of the first category. The first noise 330 corresponding to the at least one image block 320 of the first category may be determined based on the feature representation. The self-attention mechanism is used to compute correlations between elements within a sequence (that is, image units in the at least one image block 320 of the first category), and dynamically adjust a degree of attention to each element based on the correlations, thereby capturing a complex dependency inside the sequence.

[0055] In some implementations, the machine learning model 140 may include one or more processing blocks (for example, transformer blocks). An input to the processing block is defined to include a query feature, a key feature, and a value feature. The processing block may perform the self-attention mechanism on the input by performing an operation on the query feature, key feature, and value feature. Because the processing block does not involve an operation across the image unit, computations may also be performed where the input is incomplete (that is, does not include the at least one image block 325 of the second category). For example, in the case that the query feature, the key feature, and the value feature corresponding to the at least one image block 325 of the second category' do not exist, the self-attention mechanism may also be applied to the at least one image block 320 of the first category;

[0056] The application of the self-attention mechanism to the at least one image block 320 of the first category' will be described below with reference to FIG. 4. FIG. 4 illustrates a schematic diagram of a process 400 of applying a self-attention mechanism to the at least one image block 320 of the first category according to some implementations of the present disclosure. As show n in FIG. 4, the processing block may include a key projection module 406, a query projectionmodule 407, a value projection module 408, a self-attention module 440, and an output projection module 450. A first key feature 410, a first query feature 411, and a first value feature 412 to be input to the processing block in the current round may be determined using the key projection module 406, the query projection module 407. and the value projection module 408, respectively, based on the at least one image block 320 (represented by Xa) of the first category. Taking the key projection module 406 determining the first key feature 410 as an example for description, in the key projection module 406, linear transformation may be performed on respective image units in the at least one image block 320 of the first category, to obtain the first key feature 410. For example, the respective image units in the at least one image block 320 of the first category may be multiplied by a weight matrix, to obtain the first key feature 410. The first key feature 410 may be in a form of a matrix, and the weight matrix may be continuously updated through training. The process of determining the first query feature 411 and the first value feature 412 may be similar to the process of determining the first key feature 410, details of which are not described herein again.

[0057] After obtaining the first key feature 410, the first query feature 411, and the first value feature 412, in the self-attention module 440, the self-attention mechanism may be applied to the at least one image block 320 of the first category based on the three features, to obtain the feature representation (not shown in the figure) of the at least one image block 320 of the first category. In the current round, applying the self-attention mechanism to the at least one image block 320 of the first category may be as follows:where Oarepresents the feature representation of the at least one image block 320 of the first category, Qarepresents the first query feature 411, Karepresents the first key feature 410, Varepresents the first value feature 412, and d represents the dimension of the first key feature 410.

[0058] As can be seen from the above Eq. (2), the at least one image block 325 of the second category is not taken into account when applying the self-attention mechanism to the at least one image block 320 of the first category, which may affect the result of the self-attention mechanism. Therefore, it may consider the influence of the at least one image block 325 of the second category on the application of the self-attention mechanism to the at least one image block 320 of the first category.

[0059] In some implementations, a second key feature 410 and a second value feature 420 of respective image blocks in the input image 305 in the previous round may be obtained. In each round, the key feature and the value feature of the respective image blocks in the input image 305 may be stored, therefore the second key feature 410 and the second value feature 420 of theprevious round may be obtained. After the second key feature 410 and the second value feature 420 are obtained, a third key feature and a third value feature (not shown) of the at least one image block 325 of the second category in the current round may be determined from the second key feature 410 and the second value feature 420 in a fusing module 415 and a fusing module 425, respectively. Taking the fusing module 415 determining the third key feature from the second key feature 410 as an example for description, the third key feature may be estimated at least according to the positional information and the size information of the at least one image block 325 of the second category. For the process of determining the third value feature, refer to the process of determining the third key feature, and details are not described herein again.

[0060] Next, a fused key feature 430 may be obtained by the fusing module 415 fusing the first key feature 410 and the third key feature. A fused value feature 435 may be obtained by the fusing module 425 fusing the first value feature 412 and the third value feature. After obtaining the fused key feature 430 and the fused value feature 435, in the self-attention module 440, the self-attention mechanism may be applied to the at least one image block 320 of the first category based on the first query feature 411, the fused key feature 430, and the fused value feature 435, to obtain a feature representation 445 of the at least one image block 320 of the first category'. Applying the self- attention mechanism to the at least one image block 320 of the first category may be as follows:<where Oarepresents the feature representation 445, Ktrepresents the third key feature,represents the third value feature,represents the fused key feature 430, and [Va' il represents the fused value feature 435. In this way, by fusing the key feature and the value feature corresponding to the at least one image block 325 of the second category, more information in the input image 305 may be captured, enabling the feature representation 445 to better characterize the at least one image block 320 of the first category', so as to have the first noise 330 predicted more accurately.

[0061] In some implementations, the feature representation 445 of the at least one image block 320 of the first category may be input to the projection module 450 to perform linear transformation on the feature representation 445, so that subsequent processing may be performed. For example, the feature representation 445 after the linear transformation may be used to determine the first noise 330 corresponding to the at least one image block 320 of the first category.

[0062] From the observations of FIG. 2, it can be seen that the correlation between the initial sampling steps (for example, from the first to the fourth step) is lower compared to the latter stages, where the diffusion process exhibits greater stability'. Consequently, the above method for classify ing the plurality of image blocks in the input image into the first category’ and the secondcategory is not suitable for the early stages of the diffusion process, as it could negatively impact the foundational structure of the generated image. Furthermore, for the initial sampling step, the present disclosure may process all image blocks in the input image 305 instead of using the noise of some image blocks for the previous round, thereby mitigating any adverse effect on the outline of the generated image. After the initial sampling step, the plurality of image blocks in the input image may be classified into the first category and the second category, gradually decreasing sampling ratios of the image blocks of the second category.

[0063] In some implementations, the current round is after the initial round (for example, from the first round to the fourth round) of the plurality of iterative rounds. For the initial round, an input image for the initial round may be obtained, and a third noise corresponding to the input image is determined in the initial round using the machine learning model 140. Next, the third noise may be removed from the input image for the initial round, to obtain an input image for a next round. In this way, a balance between computational efficiency and the image quality can be ensured, enabling effective sampling ratios while minimizing adverse impacts on the generated image.

[0064] It can be seen from some observations that if an image block is determined as the second category multiple times, in each round, the denoising direction of the image block may deviate from the correct denoising direction, thereby causing serious accumulated errors in multiple rounds. Therefore, the accumulated error needs to be periodically reset, for example, noises of all image blocks in the input image may be predicted by the machine learning model 140 instead of classify ing the input image.

[0065] In some implementations, for a specified round (for example, the twelfth round, the twentieth round, or the like) in the plurality of iterative rounds, an input image for the specified round may be obtained, and a fourth noise corresponding to the input image is determined in the specified round using the machine learning model 140. Next, the fourth noise may be removed from the input image for the specified round, to obtain an input image for a next round following the specified round. In this way. the accumulated error of the image block in the input image is periodically reset, enabling the actual denoising process to be consistent with the correct denoising direction.

[0066] Different operations performed at the plurality of iterative rounds of the machine learning model 140 will be described below with reference to FIG. 5. FIG. 5 illustrates a flowchart of a process 500 of operations performed at multiple iterative rounds of the machine learning model 140 according to some implementations of the present disclosure. As shown in FIG. 5, at block 510, for the initial round of the plurality of iterative rounds, an input image for the initial round is obtained. A noise (also referred to as the third noise) corresponding to the input image isdetermined in the initial round using the machine learning model. The noise is then removed from the input image for the initial round, to obtain an input image for a next round. In some examples, the initial round may be the first round, or the first few rounds (for example, the first 4 rounds). Since the correlation between the sampling results of the initial round in the process of sampling the image is low, it is necessary to sample the entire input image, that is, determine the noise corresponding to the entire input image, and remove the noise from the input image.

[0067] After the initial round is performed, the similarity between the sampling results of the following rounds increases, therefore the plurality of image blocks of the input image may be classified according to the importance of the plurality of image blocks, and then different processing is performed on the image blocks with different classifications. At block 520, an input image for a current round is obtained, and a plurality of image blocks in the input image are classified into a first category and a second category at least based on respective noises corresponding to the plurality of image blocks in the input image in a previous round, to obtain at least one image block of the first category and at least one image block of the second category. In some examples, respective noises corresponding to the plurality of image blocks in the input image in the previous round may be considered as an extent to which the corresponding image block needs to be changed, and the greater the noise, the more the corresponding image block needs to be changed. Thus, the input image may be classified based at least on the noise.

[0068] At block 530, a first noise corresponding to at least one image block of the first category may be determined by the machine learning model in the current round, and a second noise determined by the machine learning model for the at least one image block of the second category' in the previous round is obtained. Then, the first noise and the second noise are respectively removed from the plurality7of image blocks of the input image, to obtain an input image of a next round. In some examples, the at least one image block of the first category7corresponds to a relatively important region in the input image, and the first noise needs to be determined by the machine learning model in the current round. The at least one image block of the second category7corresponds to a relatively unimportant region in the input image, and therefore the noise of the previous round can be used.

[0069] At block 535, it is determined whether all the rounds have been performed. If all the rounds have been completed, it indicates that the machine learning model has removed all noise, and then the process 500 proceeds to block 538. At block 538, the target image generated by the machine learning model after all the rounds may be determined.

[0070] If all rounds have not been completed, the process 500 proceeds to block 540. At block 540, it is determined whether the next round is a specified round (for example, the twelfth round, the twentieth round, or the like). If an image block is determined to be of the second category inthe plurality of rounds, a serious cumulated error may result. Thus, the accumulated error needs to be reset periodically (for example, at specified rounds). In the case that the next round is not the specified round, the operations at blocks 520 and 530 are repeatedly performed.

[0071] If the next round is the specified round, the process 500 proceeds to block 550. At block 550, an input image for a specified round may be obtained, and a noise (also referred to as the fourth noise) corresponding to the input image is determined in the specified round using the machine learning model. The noise is then removed from the input image for the specified round, to obtain an input image for a next round following the specified round. After the operations at block 550 are completed, the process 500 proceeds to block 535 to continue determining whether all rounds have been completed.

[0072] In the process 500, after different processing is performed on the input image in the initial round, the round after the initial round, and the specified round, the final target image may be obtained.

[0073] FIG. 6 shows a flowchart of a process 600 of image generation based on region-adaptive sampling according to some implementations of the present disclosure. The process 600 may be implemented at the electronic device 110 of FIG. 1, for example, may be implemented with the machine learning model 140 in the electronic device 110.

[0074] As shown in FIG. 6, at block 610, the electronic device 110, in a plurality of iterative rounds of a machine learning model, obtains an input image for a cunent round, where the machine learning model is configured to generate a target image by iteratively performing denoising on input information.

[0075] At block 620, the electronic device 110 classifies a plurality of image blocks in the input image into a first category and a second category at least based on respective noises corresponding to the plurality of image blocks in the input image in a previous round, to obtain at least one image block of the first category and at least one image block of the second category.

[0076] At block 630, the electronic device 110 determines a first noise corresponding to the at least one image block of the first category by the machine learning model in the current round.

[0077] At block 640, the electronic device 110 obtains a second noise determined by the machine learning model for the at least one image block of the second category in the previous round.

[0078] At block 650, the electronic device 110 removes the first noise and the second noise respectively from the plurality of image blocks of the input image, to obtain an input image of a next round.

[0079] At block 660, the electronic device 110 determines the target image generated by the machine learning model after the plurality of iterative rounds.

[0080] In some implementations, classifying the plurality of image blocks in the input image intothe first category and the second category comprises: determining respective metrics for the plurality of image blocks at least based on the respective noises corresponding to the plurality of image blocks of the input image in the previous round, wherein a metric for an image block indicates a level of denoising demand on the image block; and classifying the plurality of image blocks in the input image into the first category and the second category based on the respective metrics for the plurality of image blocks.

[0081] In some implementations, classify ing the plurality of image blocks into the first category and the second category based on the respective metrics for the plurality of image blocks comprises: sorting the respective metrics for the plurality of image blocks in a descending order; classify ing at least one image block corresponding to at least one metric sorted prior to a specified position into the first category; and classifying at least one image block corresponding to at least one metric sorted subsequent to the specified position into the second category.

[0082] In some implementations, classifying the plurality of image blocks in the input image into the first category and the second category comprises: classifying the plurality of image blocks in the input image into the first category and the second category based on a number of times a respective image block among the plurality of image blocks is classified into the second category in historical rounds.

[0083] In some implementations, determining the first noise by the machine learning model comprises: determining, using the machine learning model, the first noise based on a round count of the current round, the at least one image block of the first category and prompt information for generating the target image.

[0084] In some implementations, determining the first noise corresponding to the at least one image block of the first category comprises: applying a self-attention mechanism to the at least one image block of the first category, to obtain a feature representation of the at least one image block of the first category; and determining, based on the feature representation, the first noise corresponding to the at least one image block of the first category.

[0085] In some implementations, the machine learning model comprises a processing block, with an input being defined to include a query feature, a key feature and a value feature, and wherein applying the self-attention mechanism to the at least one image block of the first category' comprises: determining, based on the at least one image block of the first category a first query' feature, a first key feature and a first value feature to be input to the processing block in the current round; and applying the self-attention mechanism to the at least one image block of the first category based on the first query' feature, the first key feature and the first value feature, to obtain the feature representation of the at least one image block of the first category.

[0086] In some implementations, applying the self-attention mechanism to the at least one imageblock of the first category comprises: obtaining a second key feature and a second value feature of a respective image block in the input image in the previous round; determining, from the second key feature and the second value feature, a third key feature and a third value feature of the at least one image block of the second category in the current round; and obtaining a fused key feature by fusing the first key feature and the third key feature, and a fused value feature by fusing the first value feature and the third value feature, respectively; and applying the self-attention mechanism to the at least one image block of the first category based on the first query7feature, the fused key feature, and the fused value feature, to obtain the feature representation of the at least one image block of the first category.

[0087] In some implementations, the current round is subsequent to an initial round of the plurality of iterative rounds, and the process 600 further comprises: for the initial round, obtaining an input image for the initial round; determining, using the machine learning model, a third noise corresponding to the input image in the initial round; and removing the third noise from the input image for the initial round, to obtain an input image for a next round.

[0088] In some implementations, the process 600 further comprises: for a specified round in the plurality of iterative rounds, obtaining an input image for the specified round; determining, using the machine learning model, a fourth noise corresponding to the input image in the specified round; and removing the fourth noise from the input image for the specified round, to obtain an input image for a next round following the specified round.

[0089] In some implementations, the input image comprises a feature map in a latent space or an image in a pixel space.

[0090] In some implementations, the machine learning model is constructed based on a diffusion model.

[0091] In some implementations, the at least one image block of the first category is associated with a main object in the input image, and the at least one image block of the second category is associated with a background in the input image.

[0092] FIG. 7 shows a schematic block diagram of an electronic device capable of implementing multiple implementations of the present disclosure. It should be understood that the electronic device 700 show n in FIG. 7 is merely an example and should not constitute any limitation on the functions and scope of the implementations described in the present disclosure.

[0093] As shown in FIG. 7, the electronic device 700 includes an electronic device 700 in the form of a general -purpose computing device. Components of the electronic device 700 may include, but are not limited to, one or more processors or processing devices 710, a memory 720, a storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760.

[0094] In some implementations, the electronic device 700 may be implemented as a computing device, a computing system, a server, a mainframe, or another device with computing capability.

[0095] The processing device 710 may be an actual or virtual processor and can perform various processes according to programs stored in the memory 720. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of the electronic device 700. The processing device 710 may include a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a controller, and / or a microcontroller, or the like.

[0096] The electronic device 700 typically includes a plurality of computer storage media. Such media may be any available media accessible to the electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 720 may include a volatile memory (for example, a register, a cache, a random access memory (RAM)), a non-volatile memory (for example, a read-only memory’ (ROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory), or some combination thereof. The storage device 730 may’ include a removable or non-removable medium, and may include a computer-readable medium such as a memory, a flash drive, a magnetic disk, or any other medium that can be used to store information and / or data and that can be accessed within the electronic device 700.

[0097] The electronic device 700 may further include an additional removable / non-removable or a volatile / non-volatile storage medium. Although not shown in FIG. 7, a magnetic disk drive for reading from or writing to a removable, non-volatile magnetic disk and an optical disk drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces.

[0098] The communication unit 740 enables communication with another computing device through a communication medium. In addition, the functionality7of the components of the electronic device 700 may be implemented in a single computing cluster or multiple computing machines capable of communicating over a communication connection. Thus, the electronic device 700 may operate in a networked environment using logical connections to one or more other servers, personal computers (PCs), or another general network node.

[0099] The input device 750 may be one or more various input devices, such as a mouse, a keyboard, a data import device, and the like. The output device 760 may be one or more output devices, such as a display, a data export device, or the like. The electronic device 700 may also communicate with one or more external devices (not shown) through the communication unit 740 as needed, such as a storage device, a display device, or the like, with one or more devices that enable a user to interact with the electronic device 700, or with any device (for example, a networkcard, a modem, or the like) that enables the electronic device 700 to communicate with one or more other computing devices. Such communication may be performed via an input / output (I / O) interface (not shown).

[0100] In some implementations, in addition to being integrated on a single device, some or all of the respective components of the electronic device 700 may also be provided in the form of a cloud computing architecture. In a cloud computing architecture, these components may be remotely arranged and may work together to implement the functionality described in the present disclosure. In some implementations, cloud computing provides computing, software, data access, and storage services that do not require end users to know the physical location or configuration of the system or hardware providing these services. In various implementations, cloud computing provides services over a wide area network, such as the Internet, using appropriate protocols. For example, cloud computing providers provide applications over a wide area network, and they can be accessed through a web browser or any other computing component. Software or components of the cloud computing architecture and corresponding data may be stored on a server at a remote location. Computing resources in the cloud computing environment may be consolidated at remote data center locations or they may be dispersed. Cloud computing infrastructure may provide services through shared data centers even though they appear as a single access point for users. Thus, the components and functionality described herein may be provided from a service provider at a remote location using a cloud computing architecture. Alternatively, they may be provided from a conventional server, or they may be installed directly or in another manner on a client device.

[0101] The electronic device 700 may be configured to implement image generation according to multiple implementations of the present disclosure. The memory 720 may include one or more modules having one or more program instructions that may be accessed and executed by the processing unit 710 to implement the functions of the various implementations described herein. For example, the memory 720 may include an image generation module 725 configured to perform image generation according to one or more implementations described above. As shown in FIG.7, the electronic device 700 may obtain input required for image generation through the input device 750, and may provide output of image generation, such as a predicted property, through the output device 760. In some implementations, the electronic device 700 may also receive input from other devices (not shown) via the communication unit 740.

[0102] Some example implementations of the present disclosure are listed below.

[0103] In one aspect, the present disclosure provides a computer-implemented method. The method comprises: in a plurality of iterative rounds of a machine learning model, obtaining an input image for a current round, wherein the machine learning model is configured to generate atarget image by iteratively performing denoising on input information; classifying a plurality of image blocks in the input image into a first category and a second category at least based on respective noises corresponding to the plurality of image blocks in the input image in a previous round, to obtain at least one image block of the first category and at least one image block of the second category; determining a first noise corresponding to the at least one image block of the first category by the machine learning model in the current round; obtaining a second noise determined by the machine learning model for the at least one image block of the second category' in the previous round; and removing the first noise and the second noise respectively from the plurality of image blocks of the input image, to obtain an input image of a next round; and determining the target image generated by the machine learning model after the plurality of iterative rounds.

[0104] In some implementations, classify ing the plurality of image blocks in the input image into the first category and the second category comprises: determining respective metrics for the plurality of image blocks at least based on the respective noises corresponding to the plurality of image blocks of the input image in the previous round, wherein a metric for an image block indicates a level of denoising demand on the image block; and classifying the plurality' of image blocks in the input image into the first category and the second category based on the respective metrics for the plurality of image blocks.

[0105] In some implementations, classifying the plurality of image blocks into the first category and the second category based on the respective metrics for the plurality' of image blocks comprises: sorting the respective metrics for the plurality of image blocks in a descending order; classifying at least one image block corresponding to at least one metric sorted prior to a specified position into the first category; and classifying at least one image block corresponding to at least one metric sorted subsequent to the specified position into the second category.

[0106] In some implementations, classifying the plurality of image blocks in the input image into the first category and the second category comprises: classifying the plurality’ of image blocks in the input image into the first category and the second category based on a number of times a respective image block among the plurality of image blocks is classified into the second category in historical rounds.

[0107] In some implementations, determining the first noise by the machine learning model comprises: determining, using the machine learning model, the first noise based on a round count of the current round, the at least one image block of the first category and prompt information for generating the target image.

[0108] In some implementations, determining the first noise corresponding to the at least one image block of the first category comprises: applying a self-attention mechanism to the at leastone image block of the first category, to obtain a feature representation of the at least one image block of the first category; and determining, based on the feature representation, the first noise corresponding to the at least one image block of the first category.

[0109] In some implementations, the machine learning model comprises a processing block, with an input being defined to include a query feature, a key feature and a value feature, and wherein applying the self-attention mechanism to the at least one image block of the first category comprises: determining, based on the at least one image block of the first category', a first query feature, a first key feature and a first value feature to be input to the processing block in the current round; and applying the self-attention mechanism to the at least one image block of the first category based on the first query feature, the first key feature and the first value feature, to obtain the feature representation of the at least one image block of the first category.

[0110] In some implementations, applying the self-attention mechanism to the at least one image block of the first category comprises: obtaining a second key feature and a second value feature of a respective image block in the input image in the previous round; determining, from the second key feature and the second value feature, a third key feature and a third value feature of the at least one image block of the second category' in the current round; and obtaining a fused key feature by fusing the first key feature and the third key feature, and a fused value feature by fusing the first value feature and the third value feature, respectively; and applying the self-attention mechanism to the at least one image block of the first category based on the first query feature, the fused key feature, and the fused value feature, to obtain the feature representation of the at least one image block of the first category7.[OHl] In some implementations, the current round is subsequent to an initial round of the plurality of iterative rounds, and the method further comprises: for the initial round, obtaining an input image for the initial round; determining, using the machine learning model, a third noise corresponding to the input image in the initial round; and removing the third noise from the input image for the initial round, to obtain an input image for a next round.

[0112] In some implementations, the method further comprises: for a specified round in the plurality of iterative rounds, obtaining an input image for the specified round; determining, using the machine learning model, a fourth noise corresponding to the input image in the specified round; and removing the fourth noise from the input image for the specified round, to obtain an input image for a next round following the specified round.

[0113] In some implementations, the input image comprises a feature map in a latent space or an image in a pixel space.

[0114] In some implementations, the machine learning model is constructed based on a diffusion model.

[0115] In some implementations, the at least one image block of the first category is associated with a main object in the input image, and the at least one image block of the second category is associated with a background in the input image.

[0116] In another aspect, the present disclosure provides an electronic device. The electronic device comprises: a processor; and a memory coupled to the processor and comprising instructions stored thereon, the instructions, when executed by the processor, causing the device to perform operations comprising: in a plurality of iterative rounds of a machine learning model, obtaining an input image for a current round, wherein the machine learning model is configured to generate a target image by iteratively performing denoising on input information; classifying a plurality of image blocks in the input image into a first category and a second category at least based on respective noises corresponding to the plurality of image blocks in the input image in a previous round, to obtain at least one image block of the first category and at least one image block of the second category; determining a first noise corresponding to the at least one image block of the first category by the machine learning model in the current round; obtaining a second noise determined by the machine learning model for the at least one image block of the second category in the previous round; and removing the first noise and the second noise respectively from the plurality of image blocks of the input image, to obtain an input image of a next round; and determining the target image generated by the machine learning model after the plurality of iterative rounds.

[0117] In some implementations, classifying the plurality of image blocks in the input image into the first category and the second category comprises: determining respective metrics for the plurality of image blocks at least based on the respective noises corresponding to the plurality of image blocks of the input image in the previous round, wherein a metric for an image block indicates a level of denoising demand on the image block; and classifying the plurality of image blocks in the input image into the first category and the second category based on the respective metrics for the plurality of image blocks.

[0118] In some implementations, classifying the plurality of image blocks into the first category and the second category based on the respective metrics for the plurality of image blocks comprises: sorting the respective metrics for the plurality of image blocks in a descending order; classifying at least one image block corresponding to at least one metric sorted prior to a specified position into the first category; and classifying at least one image block corresponding to at least one metric sorted subsequent to the specified position into the second category.

[0119] In some implementations, classifying the plurality of image blocks in the input image into the first category and the second category comprises: classifying the plurality of image blocks in the input image into the first category and the second category based on a number of times arespective image block among the plurality of image blocks is classified into the second category in historical rounds.

[0120] In some implementations, determining the first noise by the machine learning model comprises: determining, using the machine learning model, the first noise based on a round count of the current round, the at least one image block of the first category and prompt information for generating the target image.

[0121] In some implementations, determining the first noise corresponding to the at least one image block of the first category' comprises: applying a self-attention mechanism to the at least one image block of the first category, to obtain a feature representation of the at least one image block of the first category; and determining, based on the feature representation, the first noise corresponding to the at least one image block of the first category.

[0122] In some implementations, the machine learning model comprises a processing block, with an input being defined to include a query feature, a key feature and a value feature, and wherein applying the self-attention mechanism to the at least one image block of the first category comprises: determining, based on the at least one image block of the first category, a first query feature, a first key feature and a first value feature to be input to the processing block in the current round; and applying the self-attention mechanism to the at least one image block of the first category based on the first query feature, the first key feature and the first value feature, to obtain the feature representation of the at least one image block of the first category.

[0123] In some implementations, applying the self-attention mechanism to the at least one image block of the first category comprises: obtaining a second key feature and a second value feature of a respective image block in the input image in the previous round; determining, from the second key feature and the second value feature, a third key feature and a third value feature of the at least one image block of the second category in the current round; and obtaining a fused key feature by fusing the first key feature and the third key feature, and a fused value feature by fusing the first value feature and the third value feature, respectively; and applying the self-attention mechanism to the at least one image block of the first category based on the first query feature, the fused key feature, and the fused value feature, to obtain the feature representation of the at least one image block of the first category.

[0124] In some implementations, the current round is subsequent to an initial round of the plurality of iterative rounds, and the operations further comprise: for the initial round, obtaining an input image for the initial round; determining, using the machine learning model, a third noise corresponding to the input image in the initial round; and removing the third noise from the input image for the initial round, to obtain an input image for a next round.

[0125] In some implementations, the operations further comprise: for a specified round in theplurality of iterative rounds, obtaining an input image for the specified round; determining, using the machine learning model, a fourth noise corresponding to the input image in the specified round; and removing the fourth noise from the input image for the specified round, to obtain an input image for a next round following the specified round.

[0126] In some implementations, the input image comprises a feature map in a latent space or an image in a pixel space.

[0127] In some implementations, the machine learning model is constructed based on a diffusion model.

[0128] In some implementations, the at least one image block of the first category is associated with a main object in the input image, and the at least one image block of the second category' is associated with a background in the input image.

[0129] In yet another aspect, the present disclosure provides a computer-readable medium having stored thereon computer-executable instructions which, when executed by a device, cause the device to perform one or more example implementations of the method of the above aspect.

[0130] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, example ty pes of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-a-chip (SOCs), load programmable logic devices (CPLDs), and the like.

[0131] Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The program code may execute entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine or entirely on a remote machine or server.

[0132] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium may include one or more wire-based electrical connections, a portable computer diskette, a hard disk, a random-access memory’ (RAM),a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0133] Further, while operations are depicted in a particular order, it should be understood that such operations are required to be performed in the particular order shown or in a sequential order, or that all illustrated operations should be performed to achieve desired results. In certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the present disclosure. Certain features described in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations, alone or in any suitable sub-combination.

[0134] Although the subject matter has been described in language specific to structural features and / or method logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

CLAIMS1. A computer-implemented method, comprising:in a plurality' of iterative rounds of a machine learning model,obtaining an input image for a current round, yvherein the machine learning model is configured to generate a target image by iteratively performing denoising on input information;classifying a plurality of image blocks in the input image into a first category and a second category at least based on respective noises corresponding to the plurality of image blocks in the input image in a previous round, to obtain at least one image block of the first category and at least one image block of the second category;determining a first noise corresponding to the at least one image block of the first category by the machine learning model in the current round;obtaining a second noise determined by the machine learning model for the at least one image block of the second category in the previous round; andremoving the first noise and the second noise respectively from the plurality of image blocks of the input image, to obtain an input image of a next round; anddetermining the target image generated by the machine learning model after the plurality of iterative rounds.

2. The method of claim 1, yvherein classifying the plurality of image blocks in the input image into the first category and the second category comprises:determining respective metrics for the plurality of image blocks at least based on the respective noises corresponding to the plurality of image blocks of the input image in the previous round, wherein a metric for an image block indicates a level of denoising demand on the image block; andclassifying the plurality of image blocks in the input image into the first category and the second category based on the respective metrics for the plurality of image blocks.

3. The method of claim 2. wherein classifying the plurality of image blocks into the first category and the second category- based on the respective metrics for the plurality of image blocks comprises:sorting the respective metrics for the plurality' of image blocks in a descending order; classifying at least one image block corresponding to at least one metric sorted prior to a specified position into the first category; andclassifying at least one image block corresponding to at least one metric sorted subsequent to the specified position into the second category.

4. The method of claim 1 , wherein classifying the plurality of image blocks in the input imageinto the first category and the second category comprises:classifying the plurality of image blocks in the input image into the first category and the second category based on a number of times a respective image block among the plurality of image blocks is classified into the second category’ in historical rounds.

5. The method of claim 1. wherein determining the first noise by the machine learning model comprises:determining, using the machine learning model, the first noise based on a round count of the current round, the at least one image block of the first category and prompt information for generating the target image.

6. The method of claim 1, wherein determining the first noise corresponding to the at least one image block of the first category' comprises:applying a self-attention mechanism to the at least one image block of the first category, to obtain a feature representation of the at least one image block of the first category; and determining, based on the feature representation, the first noise corresponding to the at least one image block of the first category'.

7. The method of claim 6, wherein the machine learning model comprises a processing block, with an input being defined to include a query feature, a key feature and a value feature, and wherein applying the self-attention mechanism to the at least one image block of the first category comprises:determining, based on the at least one image block of the first category, a first query' feature, a first key feature and a first value feature to be input to the processing block in the current round; andapplying the self-attention mechanism to the at least one image block of the first category based on the first query' feature, the first key feature and the first value feature, to obtain the feature representation of the at least one image block of the first category.

8. The method of claim 7, wherein applying the self-attention mechanism to the at least one image block of the first category comprises:obtaining a second key feature and a second value feature of a respective image block in the input image in the previous round;determining, from the second key feature and the second value feature, a third key feature and a third value feature of the at least one image block of the second category in the cunent round; andobtaining a fused key feature by fusing the first key feature and the third key feature, and a fused value feature by fusing the first value feature and the third value feature, respectively; and applying the self-attention mechanism to the at least one image block of the first category’based on the first query feature, the fused key feature, and the fused value feature, to obtain the feature representation of the at least one image block of the first category.

9. The method of claim 1, wherein the current round is subsequent to an initial round of the plurality of iterative rounds, and the method further comprises:for the initial round,obtaining an input image for the initial round;determining, using the machine learning model, a third noise corresponding to the input image in the initial round; andremoving the third noise from the input image for the initial round, to obtain an input image for a next round.

10. The method of claim 1, further comprising:for a specified round in the plurality of iterative rounds,obtaining an input image for the specified round;determining, using the machine learning model, a fourth noise corresponding to the input image in the specified round; andremoving the fourth noise from the input image for the specified round, to obtain an input image for a next round following the specified round.

11. The method of claim 1, wherein the input image comprises a feature map in a latent space or an image in a pixel space.

12. The method of claim 1, wherein the machine learning model is constructed based on a diffusion model.

13. The method of claim 1, wherein the at least one image block of the first category is associated with a main object in the input image, and the at least one image block of the second category is associated with a background in the input image.

14. An electronic device, comprising:a processor; anda memory coupled to the processor and comprising instructions stored thereon, the instructions, when executed by the processor, causing the device to perform operations comprising:in a plurality of iterative rounds of a machine learning model,obtaining an input image for a current round, wherein the machine learning model is configured to generate a target image by iteratively performing denoising on input information;classifying a plurality of image blocks in the input image into a first category and a second category at least based on respective noises corresponding to the plurality of image blocks in the input image in a previous round, to obtain at least one image block of the firstcategory and at least one image block of the second category;determining a first noise corresponding to the at least one image block of the first category by the machine learning model in the current round;obtaining a second noise determined by the machine learning model for the at least one image block of the second category in the previous round; andremoving the first noise and the second noise respectively from the plurality of image blocks of the input image, to obtain an input image of a next round; anddetermining the target image generated by the machine learning model after the plurality of iterative rounds.

15. A computer program product tangibly stored in a computer storage medium and comprising computer executable instructions which, when executed by a device, cause the device to implement operations comprising:in a plurality of iterative rounds of a machine learning model,obtaining an input image for a current round, wherein the machine learning model is configured to generate a target image by iteratively performing denoising on input information;classifying a plurality of image blocks in the input image into a first category and a second category at least based on respective noises corresponding to the plurality of image blocks in the input image in a previous round, to obtain at least one image block of the first category and at least one image block of the second category;determining a first noise corresponding to the at least one image block of the first category by the machine learning model in the current round;obtaining a second noise determined by the machine learning model for the at least one image block of the second category in the previous round; andremoving the first noise and the second noise respectively from the plurality of image blocks of the input image, to obtain an input image of a next round; anddetermining the target image generated by the machine learning model after the plurality of iterative rounds.

Images