Method and system for generating a de-noised image

Abstract

A method (1100) includes feeding a noise-feature-map of the input image to a reverse diffusion model comprising one or more uniform core blocks. The method includes generating an enhanced feature-map for the noise-feature-map by, iteratively, performing reverse diffusion on the noise-feature-map using one or more uniform core blocks, wherein an output map-size of the noise-feature-map at the one or more uniform core blocks is same as an input map-size of the noise-feature-map. The method includes generating the de-noised image using the enhanced feature-map.

Description

METHOD AND SYSTEM FOR GENERATING A DE-NOISED IMAGE

[0001] The present disclosure generally relates to the field of image processing, and more particularly, relates to a method and a system for processing images using diffusion models.

[0002] Diffusion models may be used for image processing tasks such as image de-noising,　inpainting,　super-resolution,　image generation, video generation and the like. These tasks typically involve training a neural network to sequentially　de-noise　images after being blurred with　Gaussian noise. The diffusion models are trained to reverse the process of adding noise to the image. After training to convergence, the diffusion model may be used for image generation by starting with an image composed of random noise and applying the network iteratively to de-noise the image.

[0003] Typically, the diffusion models comprise a forward diffusion process and a reverse diffusion process. An objective of the diffusion model is to generate a probability distribution of a given dataset. In the forward diffusion process, gaussian noise may be added to the samples in the data set in steps. The reverse diffusion process is opposite to the forward diffusion process. The reverse diffusion process may be approximated with a Neural Network (NN) (e.g. U-Net) as illustrated in Figure 1, which illustrates a related U-Net based diffusion model, as per the related art.

[0004] Figure 2A illustrates a related diffusion model that relies on U-Net (encoder-decoder) structures and attention blocks, as per the related art. That is, such diffusion models are large in size and include an encoder-decoder structure. Figure 2B illustrates an image generated using a De-noising Diffusion Probabilistic Model (DDPM) framework from random noise distribution using the related diffusion model involving about 40 million parameters, as per the related art. Consequently, such diffusion models are difficult to realize on hardware because of computational complexity and memory requirements. Further, such diffusion models have irregular structures, which adds to the difficulty when being realized on hardware. Besides, efficient and optimum utilization of resources is not achieved as the diffusion models are not easily scalable because of complex and memory-intensive blocks such as self-attention blocks and long-skip connections. Further, diffusion models when used in devices such as devices having image and video processing capabilities (e.g. mobile devices), require higher inference time and power. Even with the use of Neural Processors (NPU), the inference time, and power are still high. Hence, optimization of resources is also not achieved as the quality of the images processed using the diffusion models as in the prior art is not satisfactory. Besides, the related diffusion models are not easily scalable.

[0005] This summary is provided to introduce a selection of concepts, in a simplified format, that are further described in the detailed description of the disclosure. This summary is neither intended to identify key or essential inventive concepts of the disclosure nor is it intended for determining the scope of the disclosure.

[0006] According to an embodiment, the present disclosure relates to a method for generating a de-noised image of an input image. The method includes feeding a noise-feature-map of the input image to a reverse diffusion model. The reverse diffusion model comprises one or more uniform core blocks. The method includes generating an enhanced feature-map for the noise-feature-map using the reverse diffusion model. The method includes iteratively performing reverse diffusion on the noise-feature-map using the one or more uniform core blocks. An output map-size of the noise-feature-map at the one or more uniform core blocks is same as an input map-size of the noise-feature map. Subsequently, the method includes generating the de-noised image using the enhanced feature-map.

[0007] According to an embodiment, the present disclosure relates to a system for generating a de-noised image of an input image. The system includes at least one processor comprising processing circuitry. The system includes memory storing at least one instruction. The at least one instruction, executed by the at least one processor individually or collectively, causes the system to feed a noise-feature-map of an input image to a reverse diffusion model comprising one or more uniform core blocks. The at least one instruction, executed by the at least one processor individually or collectively, causes the system to generate an enhanced feature-map for the noise-feature-map using a reverse diffusion model by, iteratively, performing reverse diffusion on the noise-feature-map using one or more uniform core blocks Wherein an output map-size of the noise-feature-map at the one or more uniform core blocks is same as an input map-size of the noise-feature map. The at least one instruction, executed by the at least one processor individually or collectively, causes the system to generate a de-noised image using the enhanced feature-map.

[0008] In an embodiment, a machine readable includes instructions that when executed by at least one processor of a system, causes the system to perform the method provided.

[0009] To further clarify the advantages and features of the present disclosure, a more particular description of the disclosure will be rendered by reference to specific embodiments thereof, which are illustrated in the appended drawing. It is appreciated that these drawings depict only typical embodiments of the disclosure and are therefore not to be considered limiting its scope. The disclosure will be described and explained with additional specificity and detail with the accompanying drawings.

[0010] These and other features, aspects, and advantages of the present disclosure will become better understood when the following detailed description is read with reference to the accompanying drawings in which like characters represent like parts throughout the drawings, wherein:

[0011] Figure 1illustrates a related diffusion model, as per the related art;

[0012] Figure 2Aillustrates a related diffusion model, as per the related art;

[0013] Figure 2Billustrates an image generated using a related diffusion model, as per the related art;

[0014] Figure 3illustrates an environment comprising a system for generating a de-noised image of an input image, in accordance with an embodiment of the present disclosure;

[0015] Figure 4illustrates the system for generating the de-noised image of the input image, in accordance with an embodiment of the present disclosure;

[0016] Figure 5illustrates the working of a feeding module of the system, in accordance with an embodiment of the present disclosure;

[0017] Figure 6illustrates the working of a map generating module of the system, in accordance with an embodiment of the present disclosure;

[0018] Figure 7further illustrates the working of the map generating module, in accordance with an embodiment of the present disclosure;

[0019] Figure 8illustrates the working of a neural network of the system, in accordance with an embodiment of the present disclosure;

[0020] Figure 9illustrates the working of a core block of the system, in accordance with an embodiment of the present disclosure;

[0021] Figure 10Aillustrates the working of a Multilayer Perceptron block of the map generating module, in accordance with an embodiment of the present disclosure;

[0022] Figure 10Billustrates an exemplary image generated by the system, in accordance with an embodiment of the present disclosure;

[0023] Figure 10Cillustrates a process flow for creating a higher resolution image by applying super resolution to a low-resolution image using the system, in accordance with an embodiment of the present disclosure; and

[0024] Figure 11is a flowchart illustrating a method for generating a de-noised image of an input image, in accordance with an embodiment of the present disclosure.

[0025] Further, skilled artisans will appreciate that elements in the drawings are illustrated for simplicity and may not have necessarily been drawn to scale. For example, the flow charts illustrate the method in terms of the most prominent steps involved to help to improve understanding of aspects of the present disclosure. Furthermore, in terms of the construction of the device, one or more components of the device may have been represented in the drawings by conventional symbols, and the drawings may show only those specific details that are pertinent to understanding the embodiments of the present disclosure so as not to obscure the drawings with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.

[0026] For the purpose of promoting an understanding of the principles of the disclosure, reference will now be made to the various embodiments, and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of the disclosure is thereby intended, such alterations and further modifications in the illustrated system, and such further applications of the principles of the disclosure as illustrated therein being contemplated as would normally occur to one skilled in the art to which the disclosure relates.

[0027] The term "some" or "one or more" as used herein is defined as "one", "more than one", or "all." Accordingly, the terms "more than one," "one or more" or "all" would all fall under the definition of "some" or "one or more". The terms "an embodiment", "another embodiment", "some embodiments", or "in one or more embodiments" may refer to one embodiment or several embodiments, or all embodiments. Accordingly, the term "some embodiments" is defined as meaning "one embodiment, or more than one embodiment, or all embodiments."

[0028] The terminology and structure employed herein are for describing, teaching, and illuminating some embodiments and their specific features and elements and do not limit, restrict, or reduce the spirit and scope of the claims or their equivalents. The phrase "exemplary" may refer to an example.

[0029] More specifically, any terms used herein such as but not limited to "includes," "comprises," "has," "consists," "have" and grammatical variants thereof do not specify an exact limitation or restriction and certainly do not exclude the possible addition of one or more features or elements, unless otherwise stated, and must not be taken to exclude the possible removal of one or more of the listed features and elements, unless otherwise stated with the limiting language "mush comprise" or "needs to include".

[0030] Whether or not a certain feature or element was limited to being used only once, either way, it may still be referred to as "one or more features", "one or more elements", "at least one feature", or "at least one element." Furthermore, the use of the terms "one or more" or "at least one" feature or element does not preclude there being none of that feature or element unless otherwise specified by limiting language such as "there needs to be one or more" or "one or more element is required."

[0031] Unless otherwise defined, all terms, and especially any technical and / or scientific terms, used herein may be taken to have the same meaning as commonly understood by one having ordinary skill in the art.

[0032] Figure 3 illustrates an environment 300 comprising a system 310 for generating a de-noised image 390-2 of an input image 390-1, in accordance with an embodiment of the present disclosure. In an embodiment, the input image 390-1 may be a frame of a sequence of frames forming a video segment. The system 310 is configured to generate denoised frames such as image 390-2 corresponding to each such frames such as the image 390-1 in the sequence of the frames. A time stamp, t=0, corresponds to the de-noised image 390-2. Similarly, a time stamp, t= T, corresponds to the input image 390-1.

[0033] The system 310 is communicably coupled with a capturing device 150 (interchangeably referred hereinafter as "the device" 150). The device 150 may be an electronic device capable of performing various functions including image and video processing capabilities. An example of device 150 may be a mobile phone device, a tablet and the like. In an embodiment, the device 150 may be a smartphone, a camera, or any other electronic device having one or more cameras compatible with capturing or recording images, video, etc. In an embodiment, the device 150 may include multiple layers, for example, an application layer, a file system layer, etc. The application layer may include a video player application, a gallery application, or a camera application, without departing from the scope of the present disclosure. Further, the file system layer may include a file reader, a CoDec, a frame data, and a file writer. The file reader may be configured to read a video recorded by the application layer.

[0034] In an embodiment, the system 310 is implemented and resides within the device 150. The system 310 is communicably coupled with the device 150. In an embodiment, the system 310 may be in the form of programmed instructions and may be located at distributed locations such as externally as a software application on a server or in cloud of a connected network. In an embodiment, the system 310 may be located on a server in communication with the device 150.

[0035] Figure 4 illustrates the system 310 for generating the de-noised image 390-2 of the input image 390-1, in accordance with an embodiment of the present disclosure. The system 310 includes a plurality of modules 400 including a feeding module 410, a map generating module 420 and an image generating module 430.

[0036] The feeding module 410 is configured for feeding a noise-feature-map 490M1 of the input image 390-1 to a reverse diffusion model 490. In an embodiment, the feeding module 410 is configured to extract the noise-feature-map 490M1 from the input image 390-1. The map generating module 420 is configured for generating an enhanced feature-map 490M2 for the noise-feature-map 490M1 using the reverse diffusion model 490. Subsequently, the image generating module 430 is configured to generate the de-noised image 390-2 using the enhanced feature-map 490M2.

[0037] In an embodiment, the system 310 includes a processor 304, a memory 308, a transceiver 326 and an I / O interface 328. The processor 304 may be disposed in communication with a communication network via a network interface. In an embodiment, the network interface may be the I / O interface 328. In an embodiment, the network interface may connect to the communication network to enable the connection of the system 310 with the device 150. The network interface may employ connection protocols including, without limitation, direct connect, Ethernet (e.g., twisted pair 10 / 100 / 1000 Base T), transmission control protocol / internet protocol (TCP / IP), token ring, IEEE 702.11a / b / g / n / x, etc. The communication network may include, without limitation, a direct interconnection, local area network (LAN), wide area network (WAN), wireless network (e.g., using Wireless Application Protocol), the Internet, etc. Using the network interface and the communication network, the system 310 may communicate with other devices. The network interface may employ connection protocols including, but not limited to, direct connect, Ethernet (e.g., twisted pair 10 / 100 / 1000 Base T), transmission control protocol / internet protocol (TCP / IP), token ring, IEEE 702.11a / b / g / n / x, etc.

[0038] In an embodiment, the memory 308 may be communicatively coupled to the processor 304. The memory 308 may be configured to store data, and instructions executable by the processor 304. In one embodiment, the memory 308 may be provided within the device 150. In an embodiment, the memory 308 may be provided within the system 310 being remote from the device 150. In an embodiment, the memory 308 may communicate with the processor 304 via a bus within the system 310. In an embodiment, the memory 308 may be located remotely from the processor 304 and may be in communication with the processor 304 via a network. The memory 308 may include, but is not limited to, a non-transitory computer-readable storage media, such as various types of volatile and non-volatile storage media including, but not limited to, random access memory, read-only memory, programmable read-only memory, electrically programmable read-only memory, electrically erasable read-only memory, flash memory, magnetic tape or disk, optical media and the like.

[0039] In one example, the memory 308 may include a cache or random-access memory for the processor 304. In one example, the memory 308 is separate from the processor 304, such as a cache memory of a processor, the system memory, or other memory. The memory 308 may be an external storage device or database for storing data. The memory 308 may be operable to store instructions executable by the processor 304. The functions, acts, or tasks illustrated in the figures or described may be performed by the programmed processor 304 for executing the instructions stored in the memory 308. The functions, acts, or tasks are independent of the particular type of instruction set, storage media, processor, or processing strategy and may be performed by software, hardware, integrated circuits, firmware, micro-code, and the like, operating alone or in combination. Likewise, processing strategies may include multiprocessing, multitasking, parallel processing, and the like.

[0040] In an embodiment, the plurality of modules 400 may be included within the memory 308. The plurality of modules 400 may include a set of instructions that may be executed to cause the system 310, in particular, the processor 304 of the system 310, to perform any one or more of the methods / processes disclosed herein. The plurality of modules 400 may be configured to perform the steps of the present disclosure using the data stored in the database. For instance, the plurality of modules 400 may be configured to perform the steps disclosed with reference to Figure 11.

[0041] In an embodiment, each of the plurality of modules 400 may be a hardware unit which may be outside the memory 308. Further, the memory 308 may include an operating system for performing one or more tasks of the system 310, as performed by a generic operating system. Each of the modules 400 may be in communication with one another and the processor 304.

[0042] At least one of the plurality of modules 400 may be implemented through an Artificial Intelligence (AI) model. A function associated with the AI model may be performed through the non-volatile memory, the volatile memory, and the processor 304.

[0043] The processor 304 may include one or a plurality of processors. At this time, one or a plurality of processors may be a general purpose processor, such as a central processing unit (CPU), an application processor (AP), or the like, a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an AI-dedicated processor such as a neural processing unit (NPU).

[0044] The one or a plurality of processors control the processing of the input data in accordance with a predefined operating rule or the AI model stored in the non-volatile memory and the volatile memory. The predefined operating rule or the AI model is provided through training or learning.

[0045] Here, being provided through learning means that, by applying a learning technique to a plurality of learning data, a predefined operating rule or AI model of a desired characteristic is made. The learning may be performed in a device itself in which AI according to an embodiment is performed, and / or may be implemented through a separate server / system.

[0046] The AI model may consist of a plurality of neural network layers. Each layer has a plurality of weight values and performs a layer operation through calculation of a previous layer and an operation of a plurality of weights. Examples of neural networks include, but are not limited to, Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), Generative Adversarial Networks (GAN), and deep Q-networks.

[0047] The learning technique is a method for training a predetermined target device (for example, a robot) using a plurality of learning data to cause, allow, or control the target device to make a determination or prediction. Examples of learning techniques include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0048] According to the disclosure, a method for generating a plurality of instructions may use an AI model to recommend / execute the plurality of instructions by using sensor data. The processor may perform a pre-processing operation on the data to convert it into a form appropriate for use as an input for the AI model. The AI model may be obtained by training. Here, "obtained by training" means that a predefined operation rule or AI model configured to perform a desired feature (or purpose) is obtained by training a basic artificial intelligence model with multiple pieces of training data by a training technique. The artificial intelligence model may include a plurality of neural network layers. Each of the plurality of neural network layers includes a plurality of weight values and performs neural network computation by computation between a result of computation by a previous layer and the plurality of weight values.

[0049] Reasoning prediction is a technique of logical reasoning and predicting by determining information and includes, e.g., knowledge-based reasoning, optimization prediction, preference-based planning, or recommendation.

[0050] The working and functioning of the plurality of modules 400 of the system 310 have been described in detail with reference to the following Figures.

[0051] Figure 5 illustrates the working of the feeding module 410 of the system 310, in accordance with an embodiment of the present disclosure. In an embodiment, the feeding module 410 is configured to generate a noise-feature-map 510 of the input image 390-1. The generated noise-feature-map 510 has a first map-size associated with the noise-feature-map 510 depending upon the input image 390-1. Further, the feeding module 410 includes a pre-processing module 520 configured to preprocess the input image 390-1 by converting the first map-size of the noise-feature-map 510 to correspond to a uniform core-block size of the system 310. Based on the noise-feature-map 510, the pre-processing module 520 is configured to generate the noise feature map 490M1 with a map size corresponding to the uniform core-block size.

[0052] Specifically, the feeding module 410 is configured to convert the map-size of the noise-feature-map 510 to a map size which is same as the uniform core-block size of the system 310. In an embodiment, the feeding module 410 is configured to convert the input image 390-1 into the noise feature map 490M1 of the dimensions of the uniform core-block size of the system 310. For example, the preprocessing block 520 may be configured to convert a gray scale image of channel 1 to channels N, wherein N is the input channel dimension of the neural network of the system 310. Similarly, the preprocessing block 520 may convert a color image with channel 3 to channels N and a latent feature map of channels n to channels N.

[0053] Figure 6 illustrates the working of the map generating module 420 of the system 310, in accordance with an embodiment of the present disclosure. The map generating module 420 is configured for iteratively performing reverse diffusion on the noise-feature-map 490M1 using one or more uniform core blocks 610 including core block 1, a core block 2 and a core block k. Further, the map generating module 420 is configured for maintaining a constant map-size of the noise-feature-map 490M1 while performing the reverse diffusion. In an embodiment, the map generating module 420 is configured to maintain the constant map-size of the noise-feature-maps 490M1 to correspond to the size of the uniform core-blocks 610. An output map-size of a noise-feature-map 610-2 at each of the one or more uniform core blocks 610 is same as an input map-size of the noise-feature map 610-1, which in turn is the same as the map-size of the noise-feature map 490M1.

[0054] Figure 7 further illustrates the working of the map generating module 420 of the system 310, in accordance with an embodiment of the present disclosure. The map generating module 420 is configured to perform the reverse diffusion in a plurality of time-steps (t) from an initial time-step (XT) to a final time-step (X1). Further, the map generating module 420 is configured to extract a plurality of features from the extracted noise-feature-map 490M1 at each of time stamps (T, t, 1) associated with the each of the plurality of time-steps (XT,Xt,X1).

[0055] In an embodiment, the input image 390-1 is filled with gaussian noise. The diffusion model 490 is used to de-noise the image 390-1 from a time stamp 't' to a time stamp 't-1.' The de-noising is performed iteratively while progressively refining the noise-feature-maps 490M1 pre-processed from the input image 390-1 by the pre-processing module 520 at each time step starting from the time stamp T to the time stamp '1.' Each of Neural Networks (NNs) 710-T, 710-t, 710-1 (NNs) of the diffusion model 490 performs the de-noising operation on the image 390-1 in steps. At each time step (XT,Xt,X1), corresponding NNs 710-T, 710-t, 710-1 progressively reconstruct the original data corresponding to input image 390-1 by removing the noise. The NNs 710-T, 710-t, and 710-1 are configured to condition time-embedding by providing time-stamp information to the diffusion model regarding the time step (XT,Xt,X1) associated with the de-noising operation.

[0056] Figure 8 illustrates the working of the NN 710-t of the system 310, in accordance with an embodiment of the present disclosure. In an embodiment, each of the NNs 710-T, 710-t, 710-1 (or NNs 710) is designed by cascading the one or more core blocks 790 including core blocks 790-1, 790-k. The NNs 710 in the diffusion model 490 serve the core de-noising function. For example, the NN 710-t includes core blocks 790-1, 790-k including a plurality of modulation layers 792 (including modulation layers 792-1, 792-k), a Multilayer Perceptron block ("MLP") and one final convolutional layer 794 (including convolution layers 794-1, 794-k). Each of the core blocks 790 in the NNs 710 operates at a fixed input dimension and a fixed output dimension of the maps 490M1, 490M2, 610-1, and 610-2 associated with the image 390-1. The NNs are iteratively applied to the noisy input features in the feature maps 490M1, 610-1, 610-2 starting from time step XTto the time step X1.

[0057] Figure 9 illustrates the working of the core block 790-1 of the system 310, in accordance with an embodiment of the present disclosure. Each Core block 790 accepts the input , de-noises the input at the time step t and produces a de-noised output . Each core block 790 (such as core block 790-1, 790-k) includes a plurality of modulation layers 792 (including modulation layers 792-1, 792-k). For example, the core block 790-1 includes three modulation layers 792-1 (including modulation layer #1, modulation layer #2, and modulation layer #3), a Multilayer Perceptron block ("MLP") and one final convolutional layer 794-1. Further, each modulation layer 792-1 comprises a convolutional layer 796, a scale-shifted time embedding ("scaling") and an ReLU activation function ("ReLU"), along with a skip connection. The MLP includes dense layers and ReLU activation layers for learnable time embedding associated with the de-noising operation. The MLP is configured to provide the time-stamp information for identification of the time-step (t) associated with each of the plurality of time-steps (t).

[0058] Figure 10A illustrates the working of the MLP of the map generating module 420, in accordance with an embodiment of the present disclosure. The MLP is configured to provide the time-stamp information by encoding the time-stamp information into a positional-encoding of the plurality of modulation layers 792, e.g., 792-1. Subsequently, the MLP is configured for adding fully connected layers for further refining the integration of the time-stamp information in the plurality of modulation layers 792. The system 310 uses the time stamp information to determine the precise time step (XT,Xt,X1) in which the system 310 is operating resulting in a stable progression during the reverse diffusion process. The time steps (XT,Xt,X1) may be represented as discrete integers. For embedding, the time steps (XT,Xt,X1) may be converted into a continuous representation using a fixed or learned embedding function. An exemplary method to encode the time steps (XT,Xt,X1) may be to use positional encodings such as those used in transformer models and the like. Another exemplary method may include using a learnable embedding layer, where the time step xt is passed through an embedding layer which is configured to map to a high-dimensional vector.

[0059] In an embodiment, discrete time steps may be encoded into positional encoding and subsequently, additional layers, such as fully connected layers, to further refine the time step representation.

[0060] Referring again to Figure 7, in an embodiment, the image generating module 430 includes a post processing block 798 and is configured to convert a map-size associated with the enhanced feature map 490M2 to correspond to the map-size of input image 390-1 using the post-processing block 798. Subsequently, the image generating module 430 is configured to generate the de-noised image 390-2 based upon the converted enhanced feature map.

[0061] Figure 10B illustrates an exemplary image 1010 generated by the system 310, in accordance with an embodiment of the present disclosure. In an embodiment, using a De-noising Diffusion Probabilistic Model (DDPM) framework, true data samples may be generated from random noise distribution. Subsequently, a U-Net network and the core blocks 610 of the system 310 may be trained in the DDPM framework for handwritten digit generation from the random noise. For generating the exemplary image 1010, the system 310 involves 16 channels and 32 blocks and enables reduction in a model size and hence the complexity. For example, the system 310 enables reduction in model size for the image 1010 by about 8.5 times. Further, since the system 310 maintains the uniform map-size corresponding to the uniform core-block size (for each core block), the system 310 enables efficient utilizations of resources.

[0062] Figure 10C illustrates a process flow for creating a higher resolution image 1020HR by applying super resolution to a low-resolution image 1020 using the system 310, in accordance with an embodiment of the present disclosure. Specific weights and architecture of a pretrained diffusion model are loaded into the NN 1030t. In an embodiment, the number of time-steps or sampling steps may be specified for the diffusion process of the pretrained diffusion model.

[0063] The pre-processing module 520 of the system 310 is configured to resize the image 1020 to match the uniform core-block size of the NN 1030. A random noise tensor corresponding to the uniform core-block size / resized image 1022 is created and initialized. Subsequently, iteration through the pretrained diffusion model is performed in time steps. At each time step t, the pre-processed input image 1020 is concatenated with the random noise tensor and subsequently, fed into the feature extraction module to extract intermediate feature maps 1022. Further, the intermediate feature maps 1022 are passed through the NN 1030t to generate a refined feature maps 1022t. The refined feature maps 1022t represent a de-noised version of the intermediate feature map 1022. The iteration is performed as per the specified number of time steps to generate the denoised feature map 1024. Subsequently, the image generating module is configured to post process the denoised feature map 1024 to generate the higher resolution image 1020HR. In an embodiment, the random noise tensor may be updated with the denoised higher resolution image 1020HR. The higher resolution image 1020HR is ready for display / storage.

[0064] Figure 11 is a flowchart illustrating a method 1100 for generating a de-noised image 390-2 of an input image 390-1, in accordance with an embodiment of the present disclosure. Referring to Figures 3-9 together, the method 1100 may be performed by the system 310 based on instructions retrieved from non-transitory computer-readable media. A computer-readable media may include machine-executable or computer-executable instructions to perform all or portions of the described method. The computer-readable media may be, for example, digital memories, magnetic storage media, such as magnetic disks and magnetic tapes, hard drives, or optically readable data storage media.

[0065] The method 1100 includes a series of operations shown at step 1102 through step 1106 of Figure 11. The method 1100 may be performed by the system 310 in conjunction with one or more modules 400, the details of which are explained in conjunction with Figures 3-9, and the same are not repeated here for the sake of brevity. The method 1100 begins at step 1102.

[0066] At step 1102, the method 1100 includes feeding a noise-feature-map 490M1 of the input image 390-1 to a reverse diffusion model 490. In an embodiment, the method 1100 at step 1102 further includes extracting the noise-feature-map 490M1 from the input image 390-1. Furthermore, the method 1100 at step 1102 includes preprocessing the input image 390-1 by converting map-size of the noise-feature-map 510 of the image 390-1 to correspond to the uniform core-block size.

[0067] At step 1104, the method 1100 includes generating an enhanced feature map 490M2 by iteratively performing reverse diffusion on the noise-feature-map 490M1 using one or more uniform core blocks 610. Further, the method 1100 at step 1104 includes maintaining a constant map-size of the noise-feature-map 490M1 while performing the reverse diffusion. An output map-size of a noise-feature-map 610-2 at each of the one or more uniform core blocks 610 is same as an input map-size of the noise-feature map 610-1.

[0068] In an embodiment, the method 1100 at step 1104 further includes maintaining the constant map-size of the noise-feature-maps 490M1 to correspond to the uniform core-blocks 610. Further, at step 1104, the method 1100 includes performing the reverse diffusion in a plurality of time-steps (t) from an initial time-step (XT) to a final time-step (X1). Furthermore, the method 1100 at step 1104 includes extracting a plurality of features from the extracted noise-feature-map 490M1 at time stamps associated with the each of the plurality of time-steps (t). The method 1100 at step 1104 also includes providing a time-stamp information for identification of a time-step (t) associated with each of the plurality of time-steps (t) to the plurality of modulation layers and enhancing the extracted plurality of features to generate the enhanced feature map 490M2 based upon the provided time-stamp information.

[0069] In an embodiment, the method 1100 at step 1104 includes extracting the plurality of features based upon the provided time-stamp information. Further, the method 1100 at step 1104 includes providing the time-stamp information by encoding the time-stamp information into a positional-encoding of the plurality of modulation layers 792. Subsequently, the method 1100 at step 1104 includes adding fully connected layers for further refining the integration of the time-stamp information in the plurality of modulation layers 792.

[0070] At step 1106, the method 1100 includes generating the de-noised image 390-2 using the enhanced feature map 490M2. The method 1100 at step 1106 further includes converting a map-size associated with the enhanced feature map 490M2 to correspond to the map-size of input image 390-1. Subsequently, the image generating module 420 is configured to generate the de-noised image 390-2 based upon the converted enhanced feature map.

[0071] Hence, the system 310 and the method 1100 are directed at developing core blocks for designing neural networks of the diffusion model which are hardware friendly and have low complexity. The disclosure provides a core block architecture that reduces complexity and enables hardware realizable diffusion models. Further, the core blocks operate at a fixed input and output dimensions enabling efficient resource allocation. Hence, the disclosure enables hardware realizable diffusion models. The system 310 and the method 1100 also enable improvement in perceptual quality and pixel fidelity of the generated images.

[0072] The disclosure also obviates the use of computation-heavy blocks such as self-attention blocks and memory-intensive long-skip connections. The disclosure by employing a predefined fixed structure enables bringing the diffusion models to on-device, thus enabling advanced Generative Artificial intelligence-based features on devices such as mobile devices, tablets and the like, for video and mage generation. Further, the disclosure adds to higher security and privacy while processing user data. The disclosure also enables easy scalability.

[0073] While specific language has been used to describe the disclosure, any limitations arising on account of the same are not intended. As would be apparent to a person in the art, various working modifications may be made to the method in order to implement the inventive concept as taught herein.

[0074] The drawings and the forgoing description give examples of embodiments. Those skilled in the art will appreciate that one or more of the described elements may well be combined into a single functional element. Alternatively, certain elements may be split into multiple functional elements. Elements from one embodiment may be added to another embodiment. For example, orders of processes described herein may be changed and are not limited to the manner described herein.

[0075] Moreover, the actions of any flow diagram need not be implemented in the order shown; nor do all of the acts necessarily need to be performed. Also, those acts that are not dependent on other acts may be performed in parallel with the other acts. The scope of embodiments is by no means limited by these specific examples. Numerous variations, whether explicitly given in the specification or not, such as differences in structure, dimension, and use of material, are possible. The scope of embodiments is at least as broad as given by the followingclaims.

[0076] Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments. However, the benefits, advantages, solutions to problems, and any component(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential feature or component of any or all the claims.

[0077] According to an embodiment, a method includes feeding a noise-feature-map of the input image to a reverse diffusion model. The reverse diffusion model comprises one or more uniform core blocks. The method includes generating an enhanced feature-map for the noise-feature-map using the reverse diffusion model. The method includes iteratively performing reverse diffusion on the noise-feature-map using the one or more uniform core blocks. An output map-size of the noise-feature-map at the one or more uniform core blocks is same as an input map-size of the noise-feature map. Subsequently, the method includes generating a de-noised image using the enhanced feature-map.

[0078] According to an embodiment, the feeding the noise-feature-map comprises extracting the noise-feature-map from the input image.

[0079] According to an embodiment, the extracting the noise-feature-map comprises preprocessing the input image by converting the map-size of the noise-feature-map to correspond to the one or more uniform core blocks.

[0080] According to an embodiment, the input map-size of the noise-feature-map corresponds to the one or more uniform core blocks size.

[0081] According to an embodiment, the generating the enhanced feature-map comprises performing the reverse diffusion in a plurality of time-steps from an initial time-step to a final time-step. According to an embodiment, the generating the enhanced feature-map comprises extracting, using a plurality of modulation layers associated with the one or more uniform core blocks, a plurality of features from the noise-feature-map at time stamps associated with the plurality of time-steps. According to an embodiment, the generating the enhanced feature-map comprises providing, via a Multi-Layer Perceptron (MLP) block of the one or more uniform core blocks, a time-stamp information for identification of a time-step associated with the plurality of time-steps. According to an embodiment, enhancing, based upon the provided time-stamp information, the plurality of features to generate the enhanced feature-map.

[0082] According to an embodiment, the extracting the plurality of features is based on the provided time-stamp information.

[0083] According to an embodiment, the providing the time-stamp information comprises encoding the time-stamp information into a positional-encoding of the plurality of modulation layers. According to an embodiment, the providing the time-stamp information comprises adding, subsequently, additional fully connected layers for refining an integration of the time-stamp information in the plurality of modulation layers.

[0084] According to an embodiment, the generating the de-noised image comprises converting, using a post-processing block, the map-size associated with the enhanced feature-map to correspond to the map-size of the input image. According to an embodiment, the generating the de-noised image comprises generating the de-noised image based upon the converted enhanced feature-map.

[0085] According to an embodiment, the present disclosure relates to a system for generating a de-noised image of an input image. The system includes at least one processor comprising processing circuitry. The system includes memory storing at least one instruction. The at least one instruction, executed by the at least one processor individually or collectively, causes the system to feed a noise-feature-map of an input image to a reverse diffusion model comprising one or more uniform core blocks. The at least one instruction, executed by the at least one processor individually or collectively, causes the system to generate an enhanced feature-map for the noise-feature-map using a reverse diffusion model by, iteratively, performing reverse diffusion on the noise-feature-map using one or more uniform core blocks, wherein an output map-size of the noise-feature-map at the one or more uniform core blocks is same as an input map-size of the noise-feature map. The at least one instruction, executed by the at least one processor individually or collectively, causes the system to generate a de-noised image using the enhanced feature-map.

[0086] According to an embodiment, the at least one instruction, executed by the at least one processor individually or collectively, causes the system to extract the noise-feature-map from the input image.

[0087] According to an embodiment, the at least one instruction, executed by the at least one processor individually or collectively, causes the system to preprocess the input image by converting map-size of the noise-feature-map to correspond to the one or more uniform core-blocks.

[0088] According to an embodiment, the input map-size of the noise-feature-map corresponds to the one or more uniform core-blocks size.

[0089] According to an embodiment, the at least one instruction, executed by the at least one processor individually or collectively, causes the system to perform the reverse diffusion in a plurality of time-steps from an initial time-step to a final time-step. According to an embodiment, the at least one instruction, executed by the at least one processor individually or collectively, causes the system to extract, using a plurality of modulation layers associated with the one or more uniform core blocks, a plurality of features from the noise-feature-map at time stamps associated with the plurality of time-steps. According to an embodiment, the at least one instruction, executed by the at least one processor individually or collectively, causes the system to provide, via a Multi-Layer Perceptron (MLP) block of the one or more uniform core blocks, a time-stamp information for identification of a time-step associated with the plurality of time-steps. According to an embodiment, the at least one instruction, executed by the at least one processor individually or collectively, causes the system to enhance, based upon the provided time-stamp information, the plurality of features to generate the enhanced feature-map.

[0090] According to an embodiment, the at least one instruction, executed by the at least one processor individually or collectively, causes the system to extract the plurality of features based on the provided time-stamp information.

[0091] According to an embodiment, the at least one instruction, executed by the at least one processor individually or collectively, causes the system to provide the time-stamp information by encoding the time-stamp information into a positional-encoding of the plurality of modulation layers. According to an embodiment, the at least one instruction, executed by the at least one processor individually or collectively, causes the system to provide the time-stamp information by adding, subsequently, additional fully connected layers for further refining the integration of the time-stamp information in the plurality of modulation layers.

[0092] According to an embodiment, the at least one instruction, executed by the at least one processor individually or collectively, causes the system to convert, using a post-processing block, map-size associated with the enhanced feature map to correspond to the map-size of input image. According to an embodiment, the at least one instruction, executed by the at least one processor individually or collectively, causes the system to generate the de-noised image based upon the converted enhanced feature map.

[0093] In view of the above-mentioned problems, it is advantageous to provide an improved system and method that can overcome the above-mentioned problems and limitations.

Claims

1.A method (1100) comprising:feeding (1102) a noise-feature-map of an input image to a reverse diffusion model comprising one or more uniform core blocks;generating (1104), using the reverse diffusion model, an enhanced feature-map for the noise-feature-map by performing, iteratively, reverse diffusion on the noise-feature-map using the one or more uniform core blocks,wherein an output map-size of the noise-feature-map at the one or more uniform core blocks is same as an input map-size of the noise-feature-map; andgenerating (1106) a de-noised image using the enhanced feature-map.2.The method (1100) of claim 1, wherein the feeding the noise-feature-map comprises extracting the noise-feature-map from the input image.3.The method (1100) of claim 2, wherein the extracting the noise-feature-map comprises preprocessing the input image by converting the map-size of the noise-feature-map to correspond to the one or more uniform core blocks.4.The method (1100) any one of claims 1 to 3, wherein the input map-size of the noise-feature-map corresponds to the one or more uniform core blocks size.5.The method (1100) any one of claims 1 to 4, wherein the generating the enhanced feature-map comprises:performing the reverse diffusion in a plurality of time-steps from an initial time-step to a final time-step;extracting, using a plurality of modulation layers associated with the one or more uniform core blocks, a plurality of features from the noise-feature-map at time stamps associated with the plurality of time-steps;providing, via a Multi-Layer Perceptron (MLP) block of the one or more uniform core blocks, a time-stamp information for identification of a time-step associated with the plurality of time-steps; andenhancing, based upon the provided time-stamp information, the plurality of features to generate the enhanced feature-map.6.The method (1100) of claim 5, wherein the extracting the plurality of features is based on the provided time-stamp information.7.The method (1100) any one of claims 5 to 6, wherein the providing the time-stamp information comprises:encoding the time-stamp information into a positional-encoding of the plurality of modulation layers; andadding, subsequently, additional fully connected layers for refining an integration of the time-stamp information in the plurality of modulation layers.8.The method (1100) any one of claims 1 to 7, wherein the generating the de-noised image comprises:converting, using a post-processing block, the map-size associated with the enhanced feature-map to correspond to the map-size of the input image; andgenerating the de-noised image based upon the converted enhanced feature-map.9.A system (310) comprising:at least one processor (304) comprising processing circuitry; andmemory (308) storing at least one instruction, executed by the at least one processor individually or collectively, causes the system (310) to:feed a noise-feature-map of an input image to a reverse diffusion model comprising one or more uniform core blocks;generate, using the reverse diffusion model, an enhanced feature-map for the noise-feature-map by performing, iteratively, reverse diffusion on the noise-feature-map using one or more uniform core blocks,wherein an output map-size of the noise-feature-map at the one or more uniform core blocks is same as an input map-size of the noise-feature-map; andgenerate a de-noised image using the enhanced feature-map.10.The system (310) of claim 9, wherein the at least one instruction executed by the at least one processor individually or collectively, causes the system (310) to extract the noise-feature-map from the input image.11.The system (310) of claim 10, wherein the at least one instruction executed by the at least one processor individually or collectively, causes the system (310) to preprocess the input image by converting map-size of the noise-feature-map to correspond to the one or more uniform core-blocks.12.The system (310) any one of claims 9 to 11, wherein the input map-size of the noise-feature-map corresponds to the one or more uniform core-blocks size.13.The system (310) any one of claims 9 to 12, wherein the at least one instruction executed by the at least one processor individually or collectively, causes the system (310) to:perform the reverse diffusion in a plurality of time-steps from an initial time-step to a final time-step;extract, using a plurality of modulation layers associated with the one or more uniform core blocks, a plurality of features from the noise-feature-map at time stamps associated with the plurality of time-steps;provide, via a Multi-Layer Perceptron (MLP) block of the one or more uniform core blocks, a time-stamp information for identification of a time-step associated with the plurality of time-steps; andenhance, based upon the provided time-stamp information, the plurality of features to generate the enhanced feature-map.14.The system (310) any one of claims 9 to 13, wherein the at least one instruction executed by the at least one processor individually or collectively, causes the system (310) to:convert, using a post-processing block, map-size associated with the enhanced feature map to correspond to the map-size of input image; andgenerate the de-noised image based upon the converted enhanced feature map.15.A machine readable medium comprising instructions that when executed by at least one processor of a system, causes the system to perform the method of any one of claims 1 to 8.

Images