Image generation method and apparatus, electronic device, computer-readable storage medium, and computer program product

Abstract

Provided in the present application are an image generation method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product. The method comprises: extracting from an image to be processed a multi-scale image feature of the image to be processed; performing feature conversion on the multi-scale image feature to obtain a converted feature; performing feature extraction on prompt text to obtain a text feature of the prompt text; and on the basis of the converted feature and the text feature, generating, by means of a diffusion model, a target image corresponding to the image to be processed, wherein the semantics of the target image are consistent with the semantics of the prompt text.

Description

Image generation methods, apparatuses, electronic devices, computer-readable storage media, and computer program products

[0001] Cross-references to related applications

[0002] This application is based on and claims priority to Chinese Patent Application No. 2025104798381, filed on April 16, 2025, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of image processing technology, and in particular to an image generation method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0004] With the development of technology, Artificial Intelligence Generated Content (AIGC) is being used more and more widely in production and daily life. However, the images generated by this technology are not accurate enough. Summary of the Invention

[0005] This application provides an image generation method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy of the generated target image.

[0006] The technical solution of this application embodiment is implemented as follows:

[0007] This application provides an image generation method, which is executed by an electronic device, and the method includes:

[0008] Extract multi-scale image features from the image to be processed;

[0009] The multi-scale image features are transformed to obtain transformed features;

[0010] Feature extraction is performed on the prompt text to obtain its text features;

[0011] Based on the transformation features and the text features, a target image corresponding to the image to be processed is generated through a diffusion model, wherein the semantics of the target image are consistent with the semantics of the prompt text.

[0012] This application provides an image generation apparatus, which is disposed in an electronic device and includes:

[0013] The feature extraction module is configured to extract multi-scale image features from the image to be processed;

[0014] The feature conversion module is configured to perform feature conversion on the multi-scale image features to obtain converted features;

[0015] The feature extraction module is further configured to extract features from the prompt text to obtain the text features of the prompt text;

[0016] The image generation module is configured to generate a target image corresponding to the image to be processed through a diffusion model based on the transformation features and the text features, wherein the semantics of the target image are consistent with the semantics of the prompt text.

[0017] This application provides an electronic device, the electronic device comprising:

[0018] Memory is used to store executable instructions or computer programs.

[0019] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the image generation method provided in the embodiments of this application.

[0020] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the image generation method provided in this application when executed by a processor.

[0021] This application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implements the image generation method provided in this application.

[0022] The embodiments of this application have the following beneficial effects:

[0023] In the image generation method provided in this application embodiment, multi-scale image features of the image to be processed are extracted from the image to be processed. Compared with related technologies, multi-scale image features can enrich feature expression. Feature transformation is performed on the multi-scale image features to obtain transformed features. The transformed features combine image features of multiple scales, which can improve the accuracy of the transformed features. Feature extraction is performed on the prompt text to obtain the text features of the prompt text. Based on the transformed features and text features, a target image corresponding to the image to be processed is generated through a diffusion model. The semantics of the target image are consistent with the semantics of the prompt text.

[0024] In this way, feature transformation can be performed on multi-scale image features with rich feature representation to obtain more accurate transformation features. The multi-scale image features with rich feature representation include richer detailed information and semantic information of the image to be processed. Correspondingly, the transformation features also include richer detailed information and semantic information of the image to be processed. Then, based on the transformation features and text features, a target image corresponding to the image to be processed can be generated. The target image can include richer detailed information of the image to be processed, and the semantics of the target image are consistent with the semantics of the prompt text, which can generate a more accurate target image. The diffusion model can improve the image quality of the generated target image. The image generation method provided in this application embodiment can improve the accuracy of generating the target image corresponding to the image to be processed, and can improve the image quality of the generated target image. Attached Figure Description

[0025] Figure 1 is a schematic diagram of the image generation system provided in an embodiment of this application;

[0026] Figure 2 is a schematic diagram of the structure of the electronic device provided in an embodiment of this application;

[0027] Figure 3 is a flowchart illustrating the image generation method provided in an embodiment of this application;

[0028] Figure 4 is a schematic flowchart of the image generation method provided in the embodiment of this application (II).

[0029] Figure 5 is a flowchart illustrating the image generation method provided in an embodiment of this application.

[0030] Figure 6 is a schematic flowchart of the image generation method provided in the embodiments of this application;

[0031] Figure 7 is a schematic diagram of the structure of the image generation model provided in an embodiment of this application;

[0032] Figure 8 is a schematic diagram of the structure of the image generation model provided in an embodiment of this application;

[0033] Figure 9 is a flowchart illustrating the image generation method provided in an embodiment of this application.

[0034] Figure 10 is a schematic diagram of the structure of the image generation model provided in the embodiment of this application;

[0035] Figure 11 is a schematic diagram of the image generation model provided in an embodiment of this application;

[0036] Figure 12 is a schematic diagram of the structure of the image generation model provided in the embodiment of this application;

[0037] Figure 13 is a schematic diagram of the image to be processed provided in an embodiment of this application;

[0038] Figure 14 is a schematic diagram of the target image provided in an embodiment of this application;

[0039] Figure 15 is a schematic diagram of the target image provided in an embodiment of this application;

[0040] Figure 16 is a schematic diagram of the structure of the image generation model provided in the embodiment of this application;

[0041] Figure 17 is a schematic diagram of the structure of the image generation model provided in the embodiment of this application. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0044] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0045] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0046] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0047] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0048] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0049] (1) The diffusion model is a generative model that progressively adds noise to the data through a forward process and then recovers or generates the target data (such as an image) from the noise through a reverse denoising process. The Diffusion Transformer (DiT) is an architecture of the diffusion model that combines the self-attention mechanism of the Transformer model for feature extraction and noise prediction. DiT not only utilizes the global perspective of the Transformer to improve the coherence of image generation, but also has better scalability (scaling characteristics). As the number of model parameters and computational cost increases, the quality of the generated images is significantly improved. In other words, the diffusion model is a model that combines the Transformer model and can use the Transformer's self-attention mechanism to remove noise, thereby generating images.

[0050] (2) Character customization, also known as object customization, refers to a theme-driven image generation technology. Its core lies in extracting and learning the visual features (such as texture, shape, facial features, and other identity information) of a specific object (e.g., a specific character, pet, or product) provided by the user. Based on this, combined with new prompt text or scene conditions, images of that specific object in any new scene, new pose, or performing any new action are generated. The generated new images maintain a high degree of consistency in the original object's identity features; the new images are only used for scene or action differentiation and expansion.

[0051] In other words, character customization can generate images of objects doing anything in any scene (i.e., background) based on images including the objects. That is, based on user-specified images including the characters, it is possible to generate new images of the characters doing anything in any scene, where the new images are only used for differentiation.

[0052] (3) The scalable plug-in model is a non-destructive, modular model design strategy. Without changing (i.e., freezing the weights) the basic model, additional network structures can be added via bypass to introduce new control conditions or fine-tune domain-specific knowledge, allowing users to flexibly adjust the model's capacity and performance. In some embodiments, the expressive power of the model can be improved by increasing the number of layers (depth) and dimensions (width), thereby maintaining the generalization ability of the basic model while preserving its flexibility and scalability.

[0053] (4) Multi-scale image features refer to the set of image features extracted at different spatial resolutions or receptive field scales, which are used to comprehensively capture the detailed information of different sizes and levels in the image. In other words, it means that by extracting image features at different scales, the detailed information of different sizes and levels in the image can be captured. The scale can refer to the resolution or level of detail of the image. Image features refer to the specific information extracted from the image to describe the content of the image.

[0054] Multi-scale features typically include: shallow features (or high-frequency features), which mainly contain local details such as edges, contours, corners, and high-resolution textures of the image; and deep features (or low-frequency features), which mainly contain the global layout, semantic information, and low-resolution macroscopic structure of the image. By fusing these features at different scales, a richer visual context can be provided for subsequent model processing.

[0055] (5) Image quality is an important indicator used to measure whether an image meets the requirements in terms of visual effects, information expression, and technical parameters. From a subjective perspective, image quality is judged by the user's visual perception. Subjective evaluation criteria include image sharpness, color reproduction, and contrast. From an objective perspective, it can be determined by one or more of the following: image resolution, signal-to-noise ratio, structural similarity between the target image and the image to be processed, and also includes measuring the semantic alignment between the target image and the prompt text.

[0056] During the research process, the applicant discovered the following technical problems with the relevant technology:

[0057] Technical Issue 1: With the development of science and technology, Artificial Intelligence Generated Content (AIGC) is being used more and more widely in production and daily life. In related technologies, reference images and text can be combined to generate target images. However, the image features extracted by related technologies are relatively simple, resulting in inaccurate images.

[0058] Technical Issue 2: In related technologies, an image adapter plugin (IPAdapter) can be used to transfer the style of a reference image to a target image. However, the image adapter plugin only uses a single image feature, which results in the target image not being accurate enough when performing style transfer. In cases where both the reference image and the target image contain objects, the objects in the reference image and the objects in the target image differ significantly.

[0059] Technical Issue 3: In related technologies, target images can be generated by combining a model with reference images and text. However, the training methods for these models are relatively simple, resulting in insufficient model training and low image quality of the generated target images.

[0060] This application provides an image generation method, an image generation apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the accuracy of the generated target image. The image generation system provided in this application will be described below.

[0061] Referring to Figure 1, which is a schematic diagram of the structure of the image generation system provided in the embodiment of this application, the image generation system 100 shown in Figure 1 is designed to support an image generation application. The terminal 400 is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of both.

[0062] When a user needs to generate an image, the user can upload the image to be processed to the server 200. That is, the terminal 400 can send an image generation request including the image to be processed to the server 200. The user can also select the image to be processed from multiple images displayed on the terminal 400. Thus, the terminal 400 sends an image generation request including the identifier of the image to be processed to the server 200.

[0063] The user can also enter a prompt text, and the user can also select a prompt text from multiple displayed texts. Then, the terminal 400 can send an image generation request including the prompt text to the server 200. The image generation request may include at least one of the image to be processed and the prompt text.

[0064] Upon receiving an image generation request, server 200 can generate a target image semantically consistent with the prompt text, based on the image to be processed and the prompt text. Specifically, if the image generation request only includes the image to be processed, server 200 stores the prompt text; otherwise, server 200 stores the image to be processed.

[0065] In the image generation method provided in this application embodiment, multi-scale image features of the image to be processed are extracted from the image to be processed. Compared with related technologies, multi-scale image features can enrich feature expression. Feature transformation is performed on the multi-scale image features to obtain transformed features. The transformed features combine image features of multiple scales, which can improve the accuracy of the transformed features. Feature extraction is performed on the prompt text to obtain the text features of the prompt text. Based on the transformed features and text features, a target image corresponding to the image to be processed is generated through a diffusion model. The semantics of the target image are consistent with the semantics of the prompt text.

[0066] In this way, feature transformation can be performed on multi-scale image features with rich feature representation to obtain more accurate transformation features. The multi-scale image features with rich feature representation include richer detailed information and semantic information of the image to be processed. Correspondingly, the transformation features also include richer detailed information and semantic information of the image to be processed. Then, based on the transformation features and text features, a target image corresponding to the image to be processed can be generated. The target image can include richer detailed information of the image to be processed, and the semantics of the target image are consistent with the semantics of the prompt text, which can generate a more accurate target image. The diffusion model can improve the image quality of the generated target image. The image generation method provided in this application embodiment can improve the accuracy of generating the target image corresponding to the image to be processed, and can improve the image quality of the generated target image.

[0067] The following describes the application scenarios of the image generation method provided in the embodiments of this application. The image generation method provided in the embodiments of this application can be applied to film and television scenarios, online education scenarios, advertising scenarios, virtual reality scenarios, game scenarios, online social scenarios, smart home scenarios, cloud gaming scenarios, and e-commerce scenarios, etc.

[0068] In the image generation method provided in this application embodiment, a target image with semantic consistency with the prompt text is generated by including a to-be-processed image containing an object and prompt text. The target image includes an object, and at least one of the scene and role may be different from the to-be-processed image.

[0069] The images to be processed and the target images can correspond to images from film and television scenes, online education scenes, advertising scenes, virtual reality scenes, game scenes, online social scenes, etc. For film and television scenes, objects that meet the requirements of the script can be quickly created during film and television production, reducing production costs and improving efficiency. For online education scenes, vivid and interesting objects can be created for explaining knowledge, demonstrating experiments, etc., improving learning interest and efficiency.

[0070] For advertising scenarios, customized objects can be generated based on user preferences to improve the attractiveness and conversion rate of advertisements (i.e., recommendations). For virtual reality scenarios, diverse and vivid objects can be created for virtual reality / augmented reality, enhancing the user experience, developing more natural and emotional objects, and improving the human-computer interaction experience.

[0071] For gaming scenarios, images of virtual objects performing different actions in various scenes can be quickly generated, increasing the diversity of displayed virtual objects and enhancing user immersion. For online social scenarios, users can customize virtual avatars (i.e., objects) according to their actual needs and quickly generate images of these objects performing different actions in different scenes, thus enriching online social interactions.

[0072] For smart home scenarios, the image generation method provided in this application can be applied to visual interactions of smart speakers with screens, smart TVs, or home control screens. Based on images of family members (i.e., objects in the image to be processed) and the current smart home mode (e.g., 'sleep mode', 'movie mode' as prompt text), it can quickly generate a personalized dynamic image or standby wallpaper for the virtual avatar of a family member in the corresponding mode. This not only provides a highly personalized family human-computer interaction experience, transforming real family member images into stylized virtual avatars (target images), but also effectively protects user privacy while achieving intelligent monitoring.

[0073] For cloud gaming scenarios, the image generation method provided in this application can significantly optimize the allocation of cloud rendering resources. The cloud server only needs to send a basic virtual character model or a low-resolution image to the terminal, and combine it with text commands input by the player in the game or real-time triggered game events (prompt text) to directly generate a high-definition target image of the character (object) performing specific skills in a complex game scene on the edge or terminal side. This character-customized generation method not only greatly enriches the appearance and movement diversity of characters in cloud games, but also significantly reduces the bandwidth pressure of video stream transmission from the cloud to the terminal, enhancing the player's immersion.

[0074] For e-commerce scenarios, merchants only need to provide a single image of a product (such as clothing, furniture, digital products, etc.) to be processed, combined with prompts describing different usage environments, to automatically generate a product scene display image (target image). This greatly reduces the costs of traditional e-commerce photography, such as set design and models, and enables low-cost, high-efficiency customized display of products (objects) in any scenario, thereby improving consumers' shopping experience and purchase intention.

[0075] The electronic devices provided in the embodiments of this application will be described below. The electronic devices that implement the image generation method of the embodiments of this application can be terminals, servers, or a combination of the two. The terminals can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and vehicle terminals.

[0076] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and server can be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment.

[0077] Referring to Figure 2, which is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, the electronic device shown in Figure 2 includes: at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the electronic device are coupled together through a bus system 440. It is understood that the bus system 440 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 440 in Figure 2.

[0078] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0079] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0080] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0081] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0082] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0083] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0084] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0085] Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with user interface 430;

[0086] The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0087] In some embodiments, the image generation apparatus provided in this application can be implemented in software. Figure 2 shows an image generation apparatus 455 stored in memory 450, which can be software in the form of programs and plug-ins, including the following software modules: feature extraction module 4551, feature conversion module 4552, and image generation module 4553. These modules are logically related and can therefore be arbitrarily combined or further divided according to the functions they implement. The functions of each module will be described below.

[0088] In other embodiments, the image generation apparatus provided in this application can be implemented in hardware. As an example, the image generation apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the image generation method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0089] In some embodiments, the terminal or server can implement the image generation method provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run, such as image processing applications, game applications, social applications, and artificial intelligence applications; or they can be applets that can be embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0090] The image generation method provided in the embodiments of this application is described below. As mentioned above, the electronic device that implements the image generation method of the embodiments of this application can be a terminal, a server, or a combination of the two. Referring to Figure 3, Figure 3 is a schematic flowchart of the image generation method provided in the embodiments of this application. The image generation method provided in the embodiments of this application will be described below with reference to the steps shown in Figure 3, taking the server as the execution subject as an example.

[0091] In step 101, multi-scale image features of the image to be processed are extracted from the image to be processed.

[0092] In practical applications, users can upload or select images to be processed via a terminal, and users can also input or select prompt text. In some embodiments, in response to an image generation instruction for the image to be processed, the terminal can send at least one of the image to be processed and the prompt text to a server, and the server can correspondingly receive at least one of the image to be processed and the prompt text.

[0093] In some embodiments, the terminal may be equipped with image processing applications, game applications, social applications, and artificial intelligence applications, etc. The user can determine at least one of the image to be processed and prompt text through the display interface of the corresponding application, which is not limited here.

[0094] To facilitate understanding, the following example illustrates the image to be processed and the prompt text. For instance, the terminal stores an image of an object standing on a beach, where the beach is the background included in the image to be processed. The user wants to generate an image of the object lying on the beach, and the prompt text could be "the object is lying on the beach." Essentially, the prompt text can be used to describe the object's action. Of course, depending on the actual usage requirements, the prompt text can also be used to describe the scene in which the object is located (i.e., the background) and the image style of the target image to be generated.

[0095] In some embodiments, step 101 shown in FIG3 can be implemented by an image feature extraction model. The image feature extraction model is a feature extraction model used to extract image features. The image feature extraction model can be any feature extraction model in the art that can extract image features. For example, the image feature extraction model can be an image embedding model (Sigmoid Loss for Language-Image Pre-training, SigLIP), or a visual self-supervised model (DINOv2), without limitation.

[0096] In some embodiments, multi-scale image features of the image to be processed can be extracted. That is, multi-scale image feature extraction can be performed on the image to be processed to obtain its multi-scale image features. Multi-scale image feature extraction of the image to be processed includes at least one of the following: extracting corresponding image features from images to be processed at different resolutions, where the image features corresponding to images to be processed at different resolutions are the multi-scale image features. Multi-scale image feature extraction of the image to be processed can refer to extracting image features at different levels of detail from the image to be processed; where the image features at different levels of detail are the multi-scale image features.

[0097] Multiscale image features refer to the extraction of image features at different scales to capture details of different sizes and levels in an image. Scale can refer to the resolution or level of detail of the image, and image features refer to specific information extracted from the image to be processed to describe the content of the image.

[0098] Multi-scale image features can characterize different levels of detail, global, local, and semantic information in the image to be processed. Detail information includes texture, edge, and color information, while global information includes scene layout and background information. It also includes geometric and topological features of objects and the background in the image to be processed, such as object outlines, the proportion of objects in the image, object positions, relative positions between objects and the background, and relative positions between objects and other objects.

[0099] Semantic information refers to the type of an object, descriptions of its actions, scene descriptions (corresponding to the geometric and topological features of the object and background in the image to be processed), and contextual descriptions (corresponding to relative positional relationships). Local information refers to information about specific regions in an image, which can include detailed information, spatial structure information, and semantic information corresponding to those regions.

[0100] In other words, detail information is used to focus on the micro-features of the image, such as texture, edges, and colors; global information is used to focus on the macro-features of the image, such as the overall layout and background; local information is used to focus on the features of specific regions or objects in the image; and semantic information is used to focus on the higher-level meaning of the image, such as objects, actions, and scenes. These settings can be configured according to actual usage requirements.

[0101] In some embodiments, step 101 shown in FIG3 can be implemented in multiple ways. The first implementation of step 101 is described below. Multi-scale image features include deep image features and shallow image features. Refer to FIG4. FIG4 is a flowchart of the image generation method provided in the embodiment of this application.

[0102] First, we will explain deep image features and shallow image features. Shallow image features refer to image features extracted through the shallower (corresponding to N) network layers (i.e., layers closer to the input) in the first feature extraction model. Shallow image features retain higher spatial resolution, but their corresponding receptive field is smaller, meaning they can only perceive small local areas in the image.

[0103] Shallow image features contain low-level visual or high-frequency information about an image. They characterize the microscopic details of the image to be processed, including but not limited to: image edges, contours, corners, lines, basic texture features, and local color distribution. In image generation or editing tasks, shallow image features are crucial for preserving the physical details, geometric topology, and pixel-level sharpness of the image to be processed.

[0104] Deep image features refer to image features extracted through deeper (M) network layers (i.e., layers closer to the output) in the first feature extraction model. After multiple downsampling operations (such as pooling) or feature abstraction in the first feature extraction model, deep image features typically have lower spatial resolution but a larger receptive field, even capable of covering the entire image region.

[0105] Deep image features contain high-level semantic information or low-frequency information about an image. They highly abstract the macroscopic content of the image to be processed, including but not limited to: object category attributes, key components of objects, global layout of the scene, complex contextual relationships, and abstract semantic concepts. In image generation or editing tasks, deep image features determine the overall macroscopic structure, logical rationality, and macroscopic semantic alignment with the prompt text of the generated image.

[0106] In step 1011A, deep image features of the image to be processed are extracted through M feature extraction layers in the first feature extraction model.

[0107] In some embodiments, the first feature extraction model is an image feature extraction model, which may include multiple feature extraction layers. The number of feature extraction layers can be set according to actual usage requirements. For example, the number of feature extraction layers can be 10, 20, 30, etc.

[0108] The deep image features of the image to be processed are extracted through M (i.e., the first number) feature extraction layers in the first feature extraction model. In other words, the image to be processed is input into the first feature extraction model, and after forward propagation processing through M feature extraction layers, the deep image features of the image to be processed are output.

[0109] Where M can be greater than the first quantity threshold, M can be the number of feature extraction layers, deep image features have the characteristics of rich semantic information and low spatial resolution, deep image features refer to the features close to the output layer in the image feature extraction model, in this way, accurate deep image features can be obtained.

[0110] In step 1012A, shallow image features of the image to be processed are extracted through N feature extraction layers in the first feature extraction model.

[0111] In some embodiments, shallow image features of the image to be processed are extracted through N (i.e., the second number) feature extraction layers in the first feature extraction model. That is, the image to be processed is input into the first feature extraction model, and after forward propagation processing through N feature extraction layers, the shallow image features of the image to be processed are output.

[0112] M is greater than N, where both M and N are integers greater than 0. This means the second quantity is less than the first. N can be set according to actual usage requirements; for example, N can be half the number of feature extraction layers. Shallow image features refer to features close to the input layer in the image feature extraction model. Shallow image features are used to represent low-level information (i.e., detailed information) of the image to be processed, such as edge information, texture information, and color information. Shallow image features have the characteristics of high spatial resolution and less semantic information.

[0113] In some embodiments, shallow image features and deep image features can be extracted in the same forward propagation process of the first feature extraction model. That is, after the image to be processed is input into the model and processed by the first N feature extraction layers, its output is extracted as shallow image features. Subsequently, the feature data continues to propagate in the model, and after being processed by the remaining (MN) feature extraction layers, its output is used as deep image features. This method can significantly reduce the computational cost of the model.

[0114] For example, in a smart home scenario, the shallow image features extracted from the first N layers of the image mainly capture the physical texture of the sofa material and the local highlight edges of the lights; while the deep image features extracted from the M layers abstract the global spatial layout of the living room environment and the macroscopic semantic relationship of a person sitting on the sofa. The fusion of the two ensures that the generated new image neither destroys the realistic physical details of the original room nor fails to accurately align with the new textual semantics.

[0115] In step 1013A, the deep image features and the shallow image features are determined as the multi-scale image features of the image to be processed.

[0116] After obtaining deep and shallow image features, since the deep and shallow image features correspond to different levels of detail, the deep and shallow image features can be identified as multi-scale image features of the image to be processed.

[0117] Since deep image features and shallow image features contain different levels of detail and semantic information, after obtaining deep image features and shallow image features, they can be combined (or fused / stitched) to determine the multi-scale image features of the image to be processed.

[0118] In the image generation method provided in this application embodiment, accurate multi-scale image features can be obtained by extracting image features using different numbers of feature extraction layers. The deep and shallow image features included in the multi-scale image features can complement each other. That is, the multi-scale image features can include both rich semantic information and rich detail information. The accurate multi-scale image features can be used to generate more accurate target images in the future, thereby improving the accuracy of the target image and effectively enhancing the detail realism and semantic consistency of the final generated target image.

[0119] In some embodiments, the second implementation of step 101 is described below. Continuing from steps 1011A-1012A shown in Figure 4, mid-level image features of the image to be processed can also be extracted. That is, mid-level image features of the image to be processed can be extracted through a third number (which can be set to P) of feature extraction layers in the first feature extraction model. The third number is greater than the second number and less than the first number. The third number can be set according to actual usage requirements. Mid-level image features can capture intermediate information in the image to be processed. Mid-level image features are between shallow and deep image features in terms of semantic and spatial information, possessing moderate semantic information and moderate spatial resolution.

[0120] In some embodiments, the shallow, mid-level, and deep image features correspond to different levels of detail. These features can be defined as multi-scale image features of the image to be processed. In this way, accurate multi-scale image features can be obtained. Of course, image features are not limited to shallow, mid-level, and deep image features. Depending on the actual application requirements, image features extracted through different numbers of feature extraction layers can be used. The first and second implementation methods described above can be equivalently replaced, and will not be elaborated upon here.

[0121] Combining the first and second implementation methods described above, it can be seen that multi-scale image features can include multiple image features extracted through different numbers of feature extraction layers. One image feature corresponds to one number of feature extraction layers. As the number of feature extraction layers gradually increases, the semantic information of the corresponding image feature gradually increases, while the spatial resolution gradually decreases. In other words, the number of feature extraction layers corresponding to an image feature is positively correlated with the semantic information of the image feature, and the number of feature extraction layers corresponding to an image feature is inversely correlated with the spatial resolution of the image feature. Multi-scale image features can include not only rich semantic information but also rich detail information. Accurate multi-scale image features can be used to generate more accurate target images, thereby improving the accuracy of the target image and effectively enhancing the visual realism and semantic consistency of the target image.

[0122] The third implementation method of step 101 is described below. In some embodiments, the deep image features of the image to be processed can be extracted by the first feature extraction model. At the same time or subsequently, the image to be processed is divided into multiple image blocks of the same size. For each image block, the first feature extraction model is used to extract features from the image block to obtain the image features of the image block. The number of image blocks is consistent with the number of image features of the image blocks.

[0123] In other words, deep image features of the image to be processed can be extracted using a first number (M) of feature extraction layers in the first feature extraction model. Furthermore, feature extraction can be performed on each image patch using the first number of feature extraction layers in the first feature extraction model, resulting in the image features of the corresponding image patch. Here, the image features of the image patch are equivalent to the local image features of the image to be processed, while the deep image features are equivalent to the global image features of the image to be processed. That is, deep image features possess rich global information, while the image features of the image patch possess rich local information.

[0124] In some embodiments, if the first feature extraction model has specific requirements for the input size, each image patch can be scaled or padded to the standard input resolution required by the first feature extraction model before feature extraction is performed on each image patch. Subsequently, it is processed by alignment using a first number of feature extraction layers.

[0125] In some embodiments, after obtaining the image features of image patches, the deep image features and the image features of multiple image patches can be determined as the multi-scale image features of the image to be processed. In this way, accurate multi-scale image features can be obtained by dividing the image to be processed. The deep image features and the image features of multiple image patches included in the multi-scale image features can complement each other. That is, the multi-scale image features can include both rich global information and rich local information. The accurate multi-scale image features can be used to generate more accurate target images in the future, thereby improving the accuracy of the target images and effectively enhancing the visual realism and semantic consistency of the target images.

[0126] In some embodiments, after obtaining the image features of an image patch, the deep image features, shallow image features, and image features of multiple image patches can be determined as multi-scale image features of the image to be processed. There is a correspondence between the multi-scale image features. The correspondence may include the correspondence between deep image features and shallow image features, and may also include the correspondence between deep image features and image features of multiple image patches.

[0127] The correspondence can also include spatial mapping (e.g., a local feature of an image patch corresponds to a specific region in a deep global feature in spatial coordinates) or semantic association (e.g., edge details in shallow features combine to form a semantic object in deep features). In subsequent processing, feature alignment can be performed based on the correspondence, so that multi-scale image features can include rich global semantic information.

[0128] Image features with corresponding relationships are complementary. Multi-scale image features can include not only rich semantic information but also rich detailed information. Furthermore, multi-scale image features can include not only rich global information but also rich local information. Accurate multi-scale image features can be used to generate more accurate target images, thereby improving the accuracy of target images and effectively enhancing the visual realism and semantic consistency of target images.

[0129] The fourth implementation method of step 101 is described below. The first feature extraction model is used to extract the image features corresponding to the image with a fixed pixel resolution. When the pixel resolution of the image to be processed is not a fixed pixel resolution, the pixel resolution of the image to be processed can be adjusted through the first feature extraction model to obtain the adjusted image to be processed. Then, the deep image features of the adjusted image to be processed can be extracted through the first number of feature extraction layers in the first feature extraction model.

[0130] The fixed pixel resolution can be determined based on the first feature extraction model; for example, the fixed pixel resolution can be 384 by 384. Correspondingly, shallow image features of the adjusted image to be processed can also be extracted through a second number of feature extraction layers in the first feature extraction model.

[0131] In some embodiments, the pixel resolution of the image or image patch to be processed can be adjusted by an interpolation algorithm (such as bilinear interpolation, bicubic interpolation, or nearest neighbor interpolation) or by using upsampling and downsampling networks in deep learning to obtain the fixed pixel resolution required by the first feature extraction model.

[0132] When the pixel resolution of an image patch is not fixed, the pixel resolution of the image patch can be adjusted using a first feature extraction model to obtain an adjusted image patch. The pixel resolution of the adjusted image patch is compatible with the first feature extraction model. The first feature extraction model then extracts features from the adjusted image patch to obtain its image features. "The pixel resolution of the adjusted image patch is compatible with the first feature extraction model" means that the pixel resolution of the adjusted image patch is consistent with the pixel resolution of the image that the first feature extraction model can process; in other words, the pixel resolution of the adjusted image patch is a fixed pixel resolution.

[0133] Both the adjusted image to be processed and the adjusted image patch have a fixed pixel resolution. The adjusted image patch is obtained by dividing the adjusted image to be processed; that is, the size of the adjusted image patch is smaller than the size of the adjusted image to be processed, the number of pixels per centimeter in the adjusted image patch is greater than the number of pixels per centimeter in the adjusted image to be processed, and the equivalent sampling density of the adjusted image patch on the original image content is greater than that of the adjusted image to be processed. With both inputs having a fixed pixel resolution, the adjusted image patch can present finer local details than the adjusted global image to be processed. In other words, the spatial resolution of the adjusted image patch is greater than the spatial resolution of the adjusted image to be processed. The higher the spatial resolution, the clearer the details displayed in the image at the same size.

[0134] Correspondingly, the image features of the adjusted image patch and the deep image features of the adjusted image to be processed are image features with different spatial resolutions. This is equivalent to adjusting the spatial resolution of the image patch and the image to be processed through the first feature extraction model. This achieves the extraction of differentiated features based on the local and global spatial resolutions of the image to be processed.

[0135] In some embodiments, the image features of the adjusted image patch and the deep image features of the adjusted image to be processed can be determined as the multi-scale image features of the image to be processed. The image features of the adjusted image patch have extremely high equivalent spatial resolution, thus containing extremely rich local microscopic details, while the deep image features of the adjusted image to be processed have a complete global view, thus containing rich macroscopic global information and high-level semantic information. The two complement each other in terms of feature dimensions.

[0136] Among them, the image features of the adjusted image patch not only have rich local information, but also rich semantic information. The deep image features of the adjusted image to be processed have rich semantic information. The semantic information and rich global information of the image features of the adjusted image patch are richer than the semantic information of the deep image features of the adjusted image to be processed.

[0137] In some embodiments, after obtaining the image features of the adjusted image patch, the shallow image features of the adjusted image to be processed, and the deep image features of the adjusted image to be processed, the image features of the adjusted image patch, the shallow image features of the adjusted image to be processed, and the deep image features of the adjusted image to be processed can be determined as the multi-scale image features of the image to be processed.

[0138] Among them, there are correspondences between multi-scale image features. The correspondences can include the correspondence between the shallow image features of the adjusted image to be processed and the deep image features of the adjusted image to be processed. The correspondences can also include the deep image features of the adjusted image to be processed and the correspondences between the image features of multiple adjusted image blocks.

[0139] The image generation method provided in this application can obtain more accurate multi-scale image features by dividing the image to be processed and combining the characteristics of the first feature extraction model. The deep image features and the image features of multiple image blocks included in the multi-scale image features can complement each other. That is, the multi-scale image features include rich detail information, rich global information, rich local information and rich semantic information. The more accurate multi-scale image features can be used to generate more accurate target images in the future, thereby further improving the accuracy of the target images.

[0140] In some embodiments, when the pixel resolution of the image to be processed is a fixed pixel resolution, the pixel resolution of the image to be processed does not need to be adjusted. Instead, the image block is obtained by dividing the image to be processed, and the pixel resolution of the image block is not a fixed pixel resolution. Therefore, the pixel resolution of the image block can be adjusted by the first feature extraction model to obtain the adjusted image block. The pixel resolution of the adjusted image block is adapted to the first feature extraction model. The first feature extraction model is used to extract features from the adjusted image block to obtain the image features of the adjusted image block.

[0141] After obtaining the image features of the adjusted image patches, the deep image features, shallow image features, and adjusted image patch features can be identified as the multi-scale image features of the image to be processed. In this way, based on the segmentation of the image to be processed and combined with the characteristics of the first feature extraction model, more accurate multi-scale image features can be obtained. These more accurate multi-scale image features can then be used to generate more accurate target images, thereby further improving the accuracy of the target images.

[0142] In some embodiments, when the pixel resolution of the image patch is fixed, the pixel resolution of the image to be processed can be adjusted using a first feature extraction model to obtain an adjusted image. Then, deep image features of the adjusted image can be extracted using a first number of feature extraction layers in the first feature extraction model. Correspondingly, shallow image features of the adjusted image can also be extracted using a second number of feature extraction layers in the first feature extraction model.

[0143] After obtaining the shallow image features and deep image features of the adjusted image to be processed, the deep image features, shallow image features, and image features of multiple image patches of the adjusted image to be processed can be determined as the multi-scale image features of the image to be processed.

[0144] In the fourth implementation described above, the pixel resolution of the image to be processed and the pixel resolution of the image blocks after the image to be processed are adjusted by the first feature extraction model, so that the pixel resolution of the image to be processed and the pixel resolution of the image blocks are the same, but the spatial resolution is different. In other words, the pixel resolution and spatial resolution can be adjusted by the first feature extraction model, thereby obtaining more accurate multi-scale image features. The deep image features and the image features of multiple image blocks included in the multi-scale image features can complement each other. That is, the multi-scale image features include rich global information, rich local information and rich semantic information. More accurate multi-scale image features can be used to generate more accurate target images in the future, thereby further improving the accuracy of the target image.

[0145] The fifth implementation method of step 101 is described below. Following the third and fourth implementation methods, different partitioning methods can be used to divide the image to be processed. Different partitioning methods correspond to different image blocks. That is to say, the image block can include a first image block, a second image block, a third image block, etc. In other words, multiple different partitioning granularities can be used to divide the image to be processed, thereby obtaining various types of image blocks of different sizes (e.g., a first image block of the first size, a second image block of the second size, etc.).

[0146] The first, second, and third image blocks are all different, each corresponding to a different partitioning method. The smaller the size of the partitioned image block, the higher its spatial resolution, and the clearer the details displayed at the same size. When adjusting image blocks of different sizes to the fixed pixel resolution required by the feature extraction model, the smaller the original partitioned size of the image block, the higher its equivalent sampling density (i.e., equivalent spatial resolution) on the image to be processed, thus enabling the extraction of clearer local micro-details. For each type of image block (i.e., various image blocks obtained through different partitioning methods), the third and fourth implementation methods can be used for processing; please refer to the corresponding descriptions, which will not be elaborated upon here.

[0147] In some embodiments, different partitioning methods may include: uniform partitioning according to different grid sizes (e.g., 2×2 grid, 4×4 grid); or partitioning using overlapping sliding windows; or, non-uniform partitioning based on the salient target detection results in the image to be processed, according to the bounding boxes containing different targets.

[0148] The image generation method provided in this application integrates multi-scale image features with hierarchical feature information at the global, regional (larger image patch) and detail (smaller image patch) levels to construct a richer feature pyramid representation, providing an extremely fine cross-modal alignment foundation for high-quality generation of target images.

[0149] The sixth implementation of step 101 will be described below. The images to be processed include a first image to be processed and a second image to be processed. The pixel resolution of the first image to be processed is smaller than the pixel resolution of the second image to be processed. Referring to Figure 5, which is a flowchart of the image generation method provided in this application embodiment, the sixth implementation of step 101 will be described below with reference to the steps shown in Figure 5.

[0150] The first image to be processed and the second image to be processed can be obtained by performing downsampling (or upsampling) operations on the same original input image at different ratios; or, the first image to be processed and the second image to be processed are source images obtained by image acquisition devices with different focal lengths or resolutions for the same scene.

[0151] In step 1011B, image features of the first image to be processed are extracted using the second feature extraction model. In step 1012B, image features of the second image to be processed are extracted using the second feature extraction model. In step 1013B, the image features of the first image to be processed and the image features of the second image to be processed are determined as multi-scale image features of the image to be processed.

[0152] The following describes steps 1011B-1013B. Both the second feature extraction model and the first feature extraction model are image feature extraction models. The second feature extraction model and the first feature extraction model can be the same or different, and can be set according to actual usage requirements.

[0153] The second feature extraction model is used to extract image features from the first image to be processed, and the second feature extraction model is used to extract image features from the second image to be processed. When the second feature extraction model is the same as the first feature extraction model, the first image to be processed can be extracted using a first number of feature extraction layers in the first feature extraction model, and the second image to be processed can also be extracted using the same first number of feature extraction layers.

[0154] In this study, both the image features of the first and second images to be processed are deep image features. The difference between them lies in the following: the image features of the high-resolution second image contain extremely rich fine-grained visual details (such as high-frequency information like texture and edges) and a more accurate representation of spatial structure; while the image features of the low-resolution first image filter out some redundant details, focusing more on the global macroscopic structure and basic semantic information of the image. The two complement each other in feature representation at multiple scales and have a spatial and semantic mapping relationship.

[0155] In other words, the difference between the two is that the image features of the second image to be processed are richer in detail, while the image features of the first image to be processed are less detailed. The image features of the second image to be processed have more diverse feature representations compared to the image features of the first image to be processed. The semantic information of the image features of the second image to be processed is also richer than that of the image features of the first image to be processed. There is a correspondence between the image features of the first image to be processed and the image features of the second image to be processed.

[0156] After obtaining the image features of the first image to be processed and the image features of the second image to be processed, these features are extracted from the images with different resolutions. These image features can be defined as the multi-scale image features of the image to be processed. By extracting corresponding image features from images with different resolutions, accurate multi-scale image features can be obtained. These multi-scale image features can include semantic information of varying richness. Accurate multi-scale image features can be used to generate more accurate target images, thereby improving the accuracy of the target images and effectively enhancing their visual realism and semantic consistency.

[0157] In some embodiments, the sixth implementation can be combined with the first or second implementation. For example, the image features of the first image to be processed, the image features of the second image to be processed, the deep image features, and the shallow image features can be determined as the multi-scale image features of the image to be processed. In this case, the deep image features and the shallow image features are obtained by feature extraction based on the features of the first image to be processed.

[0158] The sixth implementation can also be combined with any of the third to fifth implementations. For example, the image features of the first image to be processed, the image features of the second image to be processed, the image features of the image patch, and the deep image features can be determined as the multi-scale image features of the image to be processed. Another example is that the image features of the first image to be processed, the image features of the second image to be processed, the image features of the image patch, the shallow image features, and the deep image features can be determined as the multi-scale image features of the image to be processed. In this case, the deep image features, the image features of the image patch, and the shallow image features are obtained by feature extraction based on the features of the first image to be processed.

[0159] The image generation method provided in this application introduces multi-resolution sources, enabling the extraction of feature representations across multiple physical scales. These multi-scale image features not only encompass the stable global semantic structure at low resolution but also incorporate the extreme visual details at high resolution. This precise multi-scale feature fusion provides stronger conditional guidance for subsequent image generation, effectively improving the high-definition visual realism and cross-modal semantic consistency of the final target image.

[0160] The seventh implementation method of step 101 is described below. Following the sixth implementation method, the number of images to be processed with different resolutions can be multiple, that is, the number of images to be processed with different resolutions can be greater than or equal to 2. For each image to be processed corresponding to a resolution, the corresponding image features can be extracted. Please refer to the description of the sixth implementation method, which will not be repeated here.

[0161] The eighth implementation method of step 101 will be explained below. In order to further improve the accuracy of multi-scale image features based on the first to seventh implementation methods, multiple image feature extraction models can be used to extract the image features of the image to be processed. That is to say, the first to seventh implementation methods can each have multiple image feature extraction models to extract the corresponding image features.

[0162] In some embodiments, multiple different image feature extraction models may include convolutional neural network-based models and visual transducer-based models. Since convolutional neural network-based models are good at capturing local translation-invariant features (such as high-frequency textures), while visual transducer-based models are good at capturing global long-distance dependencies (such as macroscopic semantics) through self-attention mechanisms, the combined use of these two heterogeneous models can achieve optimal complementary effects in terms of local details and global perspectives when extracting multi-scale image features.

[0163] Referring to Figure 6, which is a flowchart of the image generation method provided in this application embodiment, the eighth implementation of step 101 will be described below with reference to the steps shown in Figure 6. The first to seventh implementations can all be equivalently replaced with the content shown in Figure 6.

[0164] In step 1011C, a first image feature of the image to be processed is extracted from the image to be processed using a third feature extraction model. This first image feature includes multiple scales. In step 1012C, a second image feature of the image to be processed is extracted from the image to be processed using a fourth feature extraction model. This second image feature also includes multiple scales. In step 1013C, the first image feature and the second image feature are fused to obtain the multi-scale image feature of the image to be processed.

[0165] In some embodiments, the first feature extraction model may be the same as the third feature extraction model, and correspondingly, the fourth feature extraction model may be different from the first feature extraction model.

[0166] The third feature extraction model can be called the main image feature extraction model, and the fourth feature extraction model can be an auxiliary feature extraction model. For ease of understanding, the eighth implementation method will be explained below with examples.

[0167] For example, the third feature extraction model can be SigLIP, and the fourth feature extraction model can be DINOv2. The third feature extraction model extracts the first image features of the image to be processed. These first image features include multiple scales; these multi-scale first image features constitute the first multi-scale image features. Essentially, the third feature extraction model can perform multi-scale image feature extraction on the image to be processed, obtaining the first multi-scale image features. These first multi-scale image features can be any of the first through seventh implementation methods mentioned above, which will not be elaborated upon here. SigLIP excels at capturing objects in fine-grained images of the image to be processed.

[0168] The fourth feature extraction model extracts the second image features of the image to be processed. These second image features include multiple scales; these multi-scale features are called second multi-scale image features. Essentially, the fourth feature extraction model can be used to extract multi-scale image features from the image to be processed, yielding the second multi-scale image features. These second multi-scale image features can correspond to any of the implementation methods mentioned in the first to seventh methods, which will not be elaborated upon here. DINOv2 can reduce the interference of the background on the object in the image to be processed.

[0169] As an image embedding model based on contrastive learning, SigLIP has advantages in visual and linguistic cross-modal alignment and is good at capturing fine-grained object features in the image to be processed that are highly related to the semantics of the prompt text. Meanwhile, DINOv2, as a visual self-supervised model, has a powerful dense feature representation capability and can effectively distinguish foreground objects from complex backgrounds, reducing the interference of the background on the feature extraction of the target object.

[0170] After obtaining the first and second multi-scale image features, the first and second image features are fused to obtain the multi-scale image features of the image to be processed. In other words, the first and second multi-scale image features can be fused to obtain the multi-scale image features of the image to be processed. If neither the first nor the second image features include image patch features, for the same scale, the image features corresponding to that scale in the first image features are concatenated or weighted and summed with the image features corresponding to that scale in the second image features to obtain the multi-scale image features of the image to be processed.

[0171] When both the first and second image features include image patch features, alignment and fusion can be performed based on the spatial location information of the image patches in the image to be processed. For image patches with the same spatial location, the image patch features extracted by the third feature extraction model are concatenated or weighted and summed with the image patch features extracted by the fourth feature extraction model at the same location, thereby obtaining multi-scale image features of the image to be processed while preserving the local spatial topology.

[0172] For image features of multiple image patches, the image features corresponding to the third feature extraction model can be fused to obtain the fused features corresponding to the third feature extraction model. Similarly, the image features corresponding to the fourth feature extraction model can be fused to obtain the fused features corresponding to the fourth feature extraction model. Then, for the same scale, the image features corresponding to that scale in the first image features can be concatenated or weighted and summed with the image features corresponding to that scale in the second image features to obtain the multi-scale image features of the image to be processed. For example, the fused features corresponding to the third feature extraction model can be fused with the fused features corresponding to the fourth feature extraction model to obtain the multi-scale image features of the image to be processed.

[0173] In this context, "same scale" refers to image features extracted by different image feature extraction models using the same method. "Same scale" also means that the image features extracted by different image feature extraction models correspond to each other in terms of equivalent spatial resolution or receptive field level. For example, the deep image features extracted by the third feature extraction model have the same scale as the deep image features extracted by the fourth feature extraction model.

[0174] The eighth implementation method allows for the use of different feature extraction models to extract multi-scale image features from the image to be processed. Different feature extraction models can improve the representation of features in certain aspects. By fusing the features extracted by different feature extraction models, the representation of multi-scale image features in multiple aspects can be further enhanced, the multi-scale image features can be further enriched, and the accuracy of multi-scale image features can be improved. Multi-scale image features can be used to generate more accurate target images in the future, thereby improving the accuracy of target images and effectively enhancing the visual realism and semantic consistency of target images.

[0175] In some embodiments, the image to be processed may include an object and a background. Before executing one or more of the first to eighth implementations described above, the image to be processed may be masked to obtain an image containing only the object. Then, one or more of the first to eighth implementations described above are executed on the image containing only the object, thereby obtaining multi-scale image features of the object. In some embodiments, the object and background in the image to be processed may also be segmented, and the transparency of the background may be set to 0. Then, one or more of the first to eighth implementations described above are executed on the image to be processed after adjusting the transparency, thereby obtaining multi-scale image features of the object.

[0176] Referring to Figure 3, in step 102, feature transformation is performed on the multi-scale image features to obtain transformed features.

[0177] Depending on the different scenarios of multi-scale image features, step 102 can be implemented in various ways, which will be described below. In some embodiments, following the first implementation method described above, deep image features can be used as query vectors, and shallow image features can be used as key vectors and value vectors, respectively. Attention calculation is then performed on the query vector, key vector, and value vector to obtain the transformed features. The method of attention calculation is shown in formula (1):

[0178] Where Q is the query vector, K is the key vector, V is the value vector, · represents the dot product, and d k It can be represented as a scaling factor, d k Let be the dimension of the key vector. This method allows for the acquisition of accurate transformation features, which are then used to generate the target image, improving the accuracy of the generated image.

[0179] Referring to Figure 7, which is a schematic diagram of the structure of the image generation model provided in the embodiment of this application, the first feature extraction model 701 is used to execute the first implementation method, that is, the image to be processed is input into the first feature extraction model 701, and deep image features and shallow image features can be obtained.

[0180] The deep image features and shallow image features are input into the feature transformation sub-model 7021 of the feature processing model 702. The feature transformation sub-model 7021 can be a Transformer. The feature transformation sub-model 7021 is used to obtain the transformed features through the above formula 1.

[0181] In some embodiments, the dimension of the features output by the feature transformation sub-model 7021 can be adjusted. In Figure 7, the dimension of the features output by the feature transformation sub-model 7021 can be adjusted by the multilayer perceptron 7022 to obtain transformed features.

[0182] In some embodiments, the multilayer perceptron 7022 can be used as a feedforward network of the feature transformation sub-model 7021. The multilayer perceptron 7022 can be used to further adjust and refine the features output by the feature transformation sub-model 7021 to obtain transformed features.

[0183] In some embodiments, the feedforward network can perform a linear transformation on the features, mapping the input features to a higher-dimensional space, increasing the expressive power of the feature processing model. Then, the linearly transformed features can be processed by a non-linear activation function to capture more complex feature relationships and obtain non-linearly activated features. The non-linearly activated features can then be linearly transformed again to map them back to their original dimension, which is the dimension corresponding to the features input to the feedforward network. This allows for the refinement and adjustment of the features.

[0184] Feedforward networks can enhance the expressive power of feature processing models. In some implementations, by mapping features to a higher-dimensional space, feedforward networks can capture more complex feature relationships. The introduction of nonlinear activation functions enables feature processing models to handle nonlinear relationships, thereby improving their expressive power.

[0185] Feedforward networks can also adjust and refine feature representations, further adjusting and refining the input feature representations to make the features richer and more accurate. This adjustment and refinement helps the feedforward network better capture complex relationships in the image to be processed. All multilayer perceptrons involved in the embodiments of this application can possess the above capabilities, and will not be elaborated further in the corresponding positions.

[0186] In some embodiments, for the second implementation described above, the number of feature transformation sub-models may be less than the number of image features obtained by the second implementation. For ease of understanding, an example is given below with reference to Figure 8. Figure 8 is a schematic diagram of the structure of the image generation model provided in the embodiment of this application. Only the feature processing model 810 in the image generation model is shown in Figure 8.

[0187] When there are image features extracted by different numbers of feature extraction layers, attention is calculated between the image features extracted by the feature extraction layer with the largest number of features and the image features extracted by each of the other feature extraction layers to obtain the corresponding transformation sub-features. There are multiple corresponding transformation sub-features. By fusing multiple corresponding transformation sub-features, the transformation feature can be obtained.

[0188] For example, in Figure 8, after obtaining the deep image feature 801, the mid-level image feature 802, and the shallow image feature 803, the deep image feature 801 and the mid-level image feature 802 can be input into the feature transformation sub-model 8101 to obtain the transformation sub-feature corresponding to the mid-level image feature 802. In some embodiments, the deep image feature 801 can be used as the query vector, and the mid-level image feature 802 can be used as the key vector and the value vector, respectively. Attention calculation is performed on the query vector, the key vector, and the value vector to obtain the transformation sub-feature corresponding to the mid-level image feature 802. Please refer to the explanation of formula (1), which will not be elaborated here.

[0189] The deep image features 801 and the shallow image features 803 can be input into the feature transformation sub-model 8102 to obtain the transformation sub-features corresponding to the shallow image features 803. In some embodiments, the deep image features 801 are used as the query vector, and the shallow image features 803 are used as the key vector and the value vector, respectively. Attention is calculated on the query vector, the key vector and the value vector to obtain the transformation sub-features corresponding to the shallow image features 803. Please refer to the explanation of formula (1), which will not be repeated here.

[0190] In the feature transformation logic, deep image features, possessing rich macroscopic semantic information, are used as query vectors, guided by high-level semantics. Mid-level or shallow image features are used as key and value vectors, respectively. This is equivalent to selectively extracting high-frequency details such as texture and edges from shallow features, guided by high-level semantics. This effectively avoids the introduction of redundant noise in shallow features, ensuring that the transformed sub-features maintain semantic correctness while enhancing the clarity of details.

[0191] After obtaining the transformation sub-features corresponding to the middle layer image feature 802 and the shallow layer image feature 803, the transformation sub-features corresponding to the middle layer image feature 802 and the shallow layer image feature 803 can be fused to obtain the transformation features.

[0192] The following example illustrates the feature fusion method. In some embodiments, the transformation sub-features corresponding to the middle layer image feature 802 and the transformation sub-features corresponding to the shallow layer image feature 803 can be concatenated to obtain the transformation feature. In other words, the feature fusion method can be concatenation.

[0193] In some embodiments, the transformed sub-features corresponding to the mid-layer image feature 802 and the transformed sub-features corresponding to the shallow image feature 803 each have corresponding weights, including a first weight and a second weight. The transformed sub-features corresponding to the mid-layer image feature 802 are multiplied by the first weight to obtain a first product, and the transformed sub-features corresponding to the shallow image feature 803 are multiplied by the second weight to obtain a second product. The first product and the second product are added together to obtain the transformed feature. That is to say, the feature fusion method can be weighted summation. Of course, depending on the actual use requirements, the feature fusion method can also be multiplication, etc., which is not limited here. All the content related to feature fusion in this application can be equivalently replaced with the above content.

[0194] In some embodiments, following the third implementation described above, see Figure 9. Figure 9 is a schematic flowchart of the image generation method provided in this application embodiment. The steps 102 shown in Figure 3 will be described below with reference to the steps shown in Figure 9.

[0195] In step 1021, attention calculation is performed on the deep image features and the shallow image features to obtain the first attention feature.

[0196] In some embodiments, attention calculation is performed on deep image features and shallow image features to obtain the first attention feature. This can be achieved by using the deep image features as the query vector and the shallow image features as the key vector and value vector, respectively. Attention calculation is then performed on the query vector, key vector, and value vector to obtain the first attention feature. For details, please refer to the corresponding descriptions, which will not be elaborated here.

[0197] In step 1022, attention calculation is performed on the deep image features and the image features of multiple image blocks to obtain the second attention features.

[0198] In some embodiments, attention calculation is performed on deep image features and image features of multiple image patches to obtain a second attention feature. This can be achieved by first fusing the image features of multiple image patches to obtain a first fused feature. The feature fusion method can be concatenation or weighted summation. After obtaining the first fused feature, the deep image features can be used as a query vector, and the first fused feature can be used as a key vector and a value vector, respectively. Attention calculation is performed on the query vector, key vector, and value vector to obtain the second attention feature. For details on formula (1), please refer to the explanation.

[0199] In some embodiments, the second attention feature can be used as a transformation feature. In this way, the accurate second attention feature can be determined and used to generate the target image in the subsequent process, which can improve the accuracy of the generated target image.

[0200] Referring to Figure 10, which is a schematic diagram of the image generation model provided in this embodiment, the image to be processed 1001 can be input into the first feature extraction model 1010 to obtain the deep image features 1004 of the image to be processed 1001. The image to be processed 1001 can be split into 4 image blocks. In Figure 10, only image block 1002 is marked. The dashed box 1003 includes 4 image blocks. The 4 image blocks included in the dashed box 1003 are input into the first feature extraction model 1010 to obtain the image features of the 4 image blocks. The image features 1005, 1006, 1007 and 1008 of the image blocks are shown in Figure 10.

[0201] After obtaining the image features 1005, 1006, 1007, and 1008 of the image block, the image features 1005, 1006, 1007, and 1008 of the image block can be concatenated to obtain the first fusion feature 1009. Then, the first fusion feature 1009 and the deep image feature 1004 can be input into the feature processing model 1020.

[0202] When stitching together image features 1005, 1006, 1007 and 1008 of image blocks, two-dimensional grid stitching can be performed according to the relative spatial position relationship of each image block in the original image to be processed, so as to preserve the spatial topology of the image blocks in the obtained first fusion feature 1009.

[0203] The feature transformation sub-model 10201 in the feature processing model 1020 can use the deep image feature 1004 as the query vector and the first fusion feature 1009 as the key vector and value vector, respectively. Attention is calculated on the query vector, key vector, and value vector to obtain the second attention feature. In this way, accurate second attention features can be obtained.

[0204] In step 1023, the first attention feature and the second attention feature are fused to obtain the transformation feature.

[0205] After obtaining the first and second attention features, they can be fused to obtain the transformation features. The feature fusion method can be concatenation or weighted summation, which can be equivalently replaced in the previous explanation and will not be elaborated further here. In this way, the transformation features can be determined by combining the first and second attention features. These transformation features are based on rich multi-scale image features, which can further improve the accuracy of the transformation features, thereby improving the accuracy of the subsequently generated target image.

[0206] The following example illustrates Figure 9. Referring to Figure 11, Figure 11 is a schematic diagram of the structure of the image generation model provided in the embodiment of this application. The image to be processed is input into the first feature extraction model 1101 to obtain the deep image features and shallow image features of the image to be processed. The image blocks after the image to be processed is divided are input into the first feature extraction model 1101 to obtain the image features of the image blocks.

[0207] Deep image features and shallow image features can be input into the feature transformation sub-model 11021 of the feature processing model 1102 to obtain the first attention feature. Refer to the description for Figure 7. The first fused feature obtained by fusing the image features of multiple image blocks and the deep image features are input into the feature transformation sub-model 11023 of the feature processing model 1102 to obtain the second attention feature. Refer to the description for Figure 10. The first attention feature and the second attention feature can then be fused to obtain the transformed feature.

[0208] In some embodiments, the dimension of the features output by the feature transformation sub-model 11021 can be adjusted. In Figure 11, the dimension of the features output by the feature transformation sub-model 11021 can be adjusted by the multilayer perceptron 11022 to obtain the first attention feature. The dimension of the features output by the feature transformation sub-model 11023 can be adjusted. In Figure 11, the dimension of the features output by the feature transformation sub-model 11023 can be adjusted by the multilayer perceptron 11024 to obtain the second attention feature.

[0209] In some embodiments, image patches are obtained by dividing the image to be processed. Compared with the image features of the image to be processed, the image features of multiple image patches are less capable of representing the image to be processed. To further improve the accuracy of the features, the second attention feature can be fused with the deep image features to obtain the second fused feature. Then, the second fused feature can be fused with the first attention feature to obtain the transformation feature. In this way, the second fused feature can include both the image features of the image patch and the image features of the image to be processed, which means it can represent the detailed information of both the image patch and the detailed information of the image to be processed. Therefore, the transformation feature determined by the second fused feature has higher accuracy, thereby improving the accuracy of the generated target image.

[0210] Referring to Figure 12, which is a schematic diagram of the image generation model provided in this embodiment, the deep image features 1201 and the shallow image features 1202 can be input into the feature transformation sub-model 12101 of the feature processing model 1210. The feature transformation sub-model 12101 can perform attention calculation on the deep image features 1201 and the shallow image features 1202. For the features output by the feature transformation sub-model 12101, the dimension of the features output by the multilayer perceptron 12102 can be adjusted to obtain the first attention feature (not shown in Figure 12).

[0211] The deep image features 1201 and the first fusion features 1203 are input into the feature transformation sub-model 12103. The feature transformation sub-model 12103 can perform attention calculation on the deep image features 1201 and the first fusion features 1203. For the features output by the feature transformation sub-model 12103, the dimension of the features output by the feature transformation sub-model 12103 can be adjusted by the multilayer perceptron 12104, thereby obtaining the second attention features (not shown in Figure 12).

[0212] For the deep image feature 1201, the dimension of the deep image feature 1201 can be adjusted by the multilayer perceptron 12105. The dimension-adjusted deep image feature 1201 is then fused with the second attention feature to obtain the second fused feature 1204. The second fused feature 1204 can then be fused with the first attention feature to obtain the transformation feature 1205.

[0213] In the image generation method provided in this application embodiment, the second fusion feature can include both the image features of the image block and the image features of the image to be processed through the feature recombination method based on the residual mechanism. As a result, the accuracy of the transformation feature determined by the second fusion feature is higher. The transformation feature can include rich detailed information, rich global information, rich local information and rich semantic information, so as to improve the accuracy of the generated target image.

[0214] In some embodiments, the fourth implementation described above can be referred to the description of FIG9, and the description of FIG9 can be equivalently replaced, and will not be repeated here. In some embodiments, following the fifth implementation described above, when the image to be processed is divided using different partitioning methods, image features of multiple image blocks can be obtained. For each image block, a corresponding first fusion feature can be obtained. Attention calculation can be performed on the first fusion feature and deep image features corresponding to each image block, and a corresponding transformation feature can be obtained. The number of corresponding transformation features is multiple. By fusing multiple corresponding transformation features, a transformation feature can be obtained.

[0215] In some embodiments, following the sixth implementation described above, the image features of the first image to be processed can be used as the query vector, and the image features of the second image to be processed can be used as the key vector and value vector, respectively; attention is calculated on the query vector, key vector, and value vector to obtain the third attention feature. The model structure is the same as in Figure 7 and will not be described further here.

[0216] When first and third attention features exist, they can be fused to obtain a fused feature. When second and third attention features exist, they can be fused to obtain a fused feature. In some embodiments, the second attention feature and deep image features can be fused first to obtain a second fused feature, and then the second fused feature and the third attention feature can be fused to obtain a fused feature. When first, second, and third attention features exist, they can be fused to obtain a fused feature.

[0217] In some embodiments, for the seventh implementation described above, the image features corresponding to the lowest resolution image to be processed can be fused with the image features corresponding to each other resolution image to be processed to obtain corresponding transformation features. There can be multiple corresponding transformation features. Fusing these multiple transformation features yields the final transformation feature. For the eighth implementation described above, please refer to the above description; further details are omitted here.

[0218] Referring to Figure 3, in step 103, feature extraction is performed on the prompt text to obtain the text features of the prompt text.

[0219] In some embodiments, the prompt text can be converted into a vector sequence, and the text features of the prompt text can be extracted from the vector sequence. The text features can capture the semantics, syntax, and contextual information of the prompt text, thereby obtaining accurate text features of the prompt text.

[0220] In some embodiments, a pre-trained text encoder can be used to segment and encode the input prompt text, thereby extracting a text feature sequence containing semantic, syntactic, and contextual information. This text feature sequence can accurately capture the user's generation intent, providing text-dimensional conditional guidance for subsequent cross-modal generation by the diffusion model.

[0221] In step 104, based on the transformation features and text features, the target image corresponding to the image to be processed is generated through a diffusion model.

[0222] In some embodiments, step 104 can be implemented by adjusting the dimension of the transformation feature to the target dimension to obtain the adjusted transformation feature, and then generating the target image corresponding to the image to be processed through a diffusion model based on the adjusted transformation feature and the text feature. The semantics of the target image are consistent with the semantics of the prompt text.

[0223] In some embodiments, the target dimension is the dimension of the latent space of the diffusion model, which is equivalent to the adjusted transformation feature being adapted to the dimension of the input feature of the diffusion model. The adjusted transformation feature can reduce the dimension compared to the transformation feature, thereby improving the speed and efficiency of generating the target image. The adjusted transformation feature can also increase the dimension compared to the transformation feature, thereby improving the utilization of computing resources. It can be set according to actual usage requirements.

[0224] In some embodiments, a vector parameter with the target dimension can be preset. This vector parameter can be used as the query vector, and the transformation features can be used as the key vector and value vector, respectively. Attention is calculated on the query vector, key vector, and value vector to obtain the adjusted transformation features, as explained in formula (1). In this way, the dimension of the transformation features can be precisely adjusted, ensuring that the adjusted transformation features can be directly applied to the diffusion model, thereby improving the accuracy of generating the target image.

[0225] For example, referring to Figure 7, the target image corresponding to the image to be processed is generated through diffusion model 703 based on transformation features and text features. As another example, referring to Figure 11, the target image corresponding to the image to be processed is generated through diffusion model 1103 based on transformation features and text features.

[0226] In some embodiments, the image to be processed includes an object and a background. Referring to FIG13, FIG13 is a schematic diagram of the image to be processed provided in an embodiment of the present application, which includes an object 1301 and a background 1302. In some embodiments, a third fused feature is obtained by fusing the adjusted transformation features and text features through a diffusion model. In some embodiments, the adjusted transformation features and text features can be input into a multilayer perceptron, so that the adjusted transformation features and text features can be fused by the multilayer perceptron to obtain the third fused feature.

[0227] In some embodiments, the adjusted transformation features and text features can be fused by concatenation or weighted summation to obtain a third fused feature. In some embodiments, the adjusted transformation features and text features can also be fused using a Transformer to obtain a third fused feature. The settings can be configured according to actual usage requirements and are not limited here.

[0228] Randomly generated image noise is used to iteratively denoise the image noise based on a third fusion feature, generating the target image corresponding to the image to be processed. The first iterative denoising method uses randomly sampled image noise from a Gaussian distribution. This image noise is used as a query vector, and the third fusion feature is used as the key vector and value vector, respectively. Attention is calculated on the query vector, key vector, and value vector to obtain a fourth attention feature. This fourth attention feature captures the relationship between the image to be processed and the prompt text.

[0229] Noise prediction is performed based on the fourth attention feature to obtain predicted noise. Based on the image noise and the predicted noise, the image noise is denoised to obtain denoised image noise. The denoised image noise is used as the query vector, and the third fusion feature is used as the key vector and value vector, respectively. Attention is calculated on the query vector, key vector, and value vector to update the fourth attention feature. The above steps are repeated, which is equivalent to iteratively denoising the image noise based on the third fusion feature, thereby generating the target image corresponding to the image to be processed.

[0230] The first iterative denoising scheme uses the third fusion feature as both the key and value vectors after each noise prediction, representing semantic consistency. This means that continuously using the third fusion feature during target image generation ensures that the generated target image maintains a high degree of semantic consistency with the input prompt text and the image to be processed. Furthermore, it preserves detail; the adjusted transformation features corresponding to the image to be processed continuously provide detailed information about the object's appearance and texture, helping the generated target image maintain a consistent visual style and detail with the input image. It also maintains stability; keeping the key and value vectors consistent during target image generation improves the stability of the image generation model (diffusion model), preventing instability in the generated results due to excessive feature changes.

[0231] The second iterative denoising method uses randomly generated image noise sampled from a Gaussian distribution. For the first denoising step, the image noise can be used as a query vector, and the third fusion feature can be used as the key vector and value vector, respectively. Attention is calculated on the query vector, key vector, and value vector to obtain the fifth attention feature. Based on the fourth attention feature, noise prediction is performed to obtain the predicted noise. Based on the image noise and the predicted noise, the image noise is denoised to obtain the denoised image noise.

[0232] Iterative denoising based on the denoised image noise can be performed by using the denoised image noise as query vector, key vector, and value vector respectively. Attention is calculated on the query vector, key vector, and value vector to obtain the sixth attention feature. The predicted noise is then updated based on the sixth attention feature. Repeating the above steps is equivalent to iteratively denoising the image noise based on the third fusion feature, thereby generating the target image corresponding to the image to be processed.

[0233] The second iterative denoising scheme uses the third fusion feature as both the key and value vectors only in the first denoising step. Subsequent denoising processes use the updated denoised image noise for attention calculations. This reduces computational complexity and improves efficiency. It also enhances the flexibility of the diffusion model, allowing it to dynamically adjust feature representations at each generation step, exploring more possible feature combinations and increasing the diversity of generated results. Furthermore, it improves feature adaptability; as the denoising process progresses, the diffusion model can gradually adjust feature representations to better adapt to the current noise level, thus generating the target image.

[0234] In some embodiments, the first iterative denoising method can be combined with the second iterative denoising method. During the iterative denoising process, the weight of the third fused feature can be gradually reduced while the weight of the predicted noise is increased. This can improve computational efficiency and the diversity of generated results while maintaining semantic consistency and detail, and can capture rich information at different scales, thereby improving the image quality of the generated target image.

[0235] In some embodiments, the first iterative denoising method can be combined with the second iterative denoising method, or a dynamic condition guidance strategy can be adopted during the time step progression of iterative denoising. That is, as the number of denoising steps increases, the weight of the third fusion feature in the cross-attention calculation is gradually reduced. This strategy of strong guidance in the early stage and weak guidance in the later stage can ensure that the target image remains highly consistent with the input conditions in terms of macroscopic semantics and physical details, while giving the diffusion model more freedom in generating high-frequency details in the later stage of generation, thereby significantly improving the visual diversity and overall image quality of the target image.

[0236] The diffusion model captures long-distance dependencies between the image to be processed and the prompt text, thereby generating high-quality target images with rich detail, making the generated target images more visually realistic and natural. The generated target images are better aligned with the input prompt text description while preserving the features of objects in the input image to be processed, ensuring the semantic accuracy and visual coherence of the generated target images.

[0237] Specifically, the target image contains at least one object or background that differs from the image to be processed. That is, the target image and the image to be processed differ in at least one of the following ways: the objects in the target image are different from those in the image to be processed; or the background in the target image is different from the background in the image to be processed.

[0238] In some embodiments, the objects included in the target image are different from the objects included in the image to be processed, the background included in the target image is different from the background included in the image to be processed, and both the objects included in the target image and the background included in the background are different from the objects included in the image to be processed.

[0239] For example, referring to Figure 14, which is a schematic diagram of a target image provided in an embodiment of this application, the target image includes an object 1401 and a background 1402. Comparing Figures 13 and 14, it can be seen that the object 1401 shown in Figure 14 is the same as the object 1301 shown in Figure 13, and the background 1402 shown in Figure 14 is different from the background 1302 shown in Figure 13. That is to say, the background of the target image is different from the image to be processed.

[0240] For example, referring to Figure 15, which is a schematic diagram of a target image provided in an embodiment of this application, the target image includes an object 1501 and a background 1502. Comparing Figures 13 and 15, it can be seen that the action of the object 1501 shown in Figure 15 is inconsistent with that of the object 1301 shown in Figure 13, and the background 1502 shown in Figure 15 is different from the background 1302 shown in Figure 13. That is to say, the background of the target image is different from the image to be processed, and the object in the target image is different from the image to be processed.

[0241] The difference between an object in the target image and an object in the image to be processed refers to one or more differences in the object's action, facial expression, clothing, hairstyle, hair accessories, or style. The object's style is the style of the target image. The objects in the target image and the objects in the image to be processed correspond to different states of the same object. The objects in the target image and the objects in the image to be processed possess object consistency, also known as role consistency. Object consistency means that the objects in the image to be processed and the objects in the target image have the same identity. In this way, object customization (i.e., role customization) can be achieved, that is, customizing the scene in which the object exists and customizing the object's actions, enabling the generation of diverse target images for the object.

[0242] In the image generation method provided in this application embodiment, multi-scale image features of the image to be processed are extracted from the image to be processed. Compared with related technologies, multi-scale image features can enrich feature expression. Feature transformation is performed on the multi-scale image features to obtain transformed features. The transformed features combine image features of multiple scales, which can improve the accuracy of the transformed features. Feature extraction is performed on the prompt text to obtain the text features of the prompt text. Based on the transformed features and text features, a target image corresponding to the image to be processed is generated through a diffusion model. The semantics of the target image are consistent with the semantics of the prompt text.

[0243] In this way, feature transformation can be performed on multi-scale image features with rich feature expression to obtain more accurate transformation features. The multi-scale image features with rich feature expression include richer detailed information and semantic information of the image to be processed. Correspondingly, the transformation features also include richer detailed information and semantic information of the image to be processed. Then, based on the transformation features and text features, a target image corresponding to the image to be processed can be generated. The target image can include richer detailed information of the image to be processed, and the semantics of the target image are consistent with the semantics of the prompt text, which can generate a more accurate target image. The diffusion model can improve the image quality of the generated target image. The image generation method provided in this application embodiment can improve the accuracy of generating the target image corresponding to the image to be processed and can improve the image quality of the generated target image, thus solving technical problems 1 and 2.

[0244] The image generation method provided in this application embodiment can use a diffusion model to generate target images. Compared with traditional convolutional networks (such as U-Net), it can effectively utilize parallel computing resources, improve the efficiency of generating target images, and the Transformer architecture is easy to extend, allowing for improvements in the quality of the generated target images by increasing the model's depth and width. The image generation scheme provided in this application embodiment can achieve object customization, that is, customizing the scene in which the object is located and customizing the object's actions, enabling the generation of diverse target images for the object.

[0245] The image generation method provided in this application implements a cross-modal image generation and editing mechanism. By combining the rich visual prior information of the image to be processed with the semantic guidance of the prompt text, and relying on the generation capability of the diffusion model, it not only retains the core features of the image to be processed (such as structure, contour, or specific style), but also accurately realizes the editing intent expressed by the user through text, thereby improving the controllability of the generated target image and enhancing the quality of the generated target image.

[0246] Multi-scale features can simultaneously capture global contextual information (coarse-grained / low-frequency features, such as overall structure and layout) and local texture details (fine-grained / high-frequency features, such as edges and pixel-level textures) of an image. This provides richer visual references when generating the target image, avoiding the loss of image details or structural distortion caused by single-scale extraction.

[0247] Feature transformation improves computational efficiency and compatibility. It effectively filters redundant information, reduces computational complexity, and accelerates the inference process of diffusion models. It enables cross-modal feature alignment, and the transformation process helps reduce modal differences between image features and subsequent text features, allowing visual features to better adapt to the input requirements of diffusion models and improving fusion results. Text features accurately capture user intent, transforming natural language prompts into high-dimensional vectors (text features) that computers can understand. This provides explicit cross-modal guidance for image generation, enabling flexible and precise control over the generated content.

[0248] To facilitate understanding of the image generation method provided in the embodiments of this application, the image generation method provided in the embodiments of this application will be illustrated below with examples. See Figure 16, which is a schematic diagram of the structure of the image generation model provided in the embodiments of this application.

[0249] In Figure 16, the image to be processed 1601 can be input into the image feature extraction model 1610. The image feature extraction model 1610 extracts features from the image to be processed 1601 to obtain feature 1 (not shown in Figure 16). The first N feature extraction layers of the image feature extraction model 1610 extract feature 2 (not shown in Figure 16) of the image to be processed 1601. Here, N is less than the total number of feature extraction layers in the image feature extraction model 1610, and N can correspond to the second number.

[0250] The image to be processed 1601 can be input into the image feature extraction model 1620. The image feature extraction model 1620 extracts features from the image to be processed 1601 to obtain feature 3 (not shown in Figure 16). The first F feature extraction layers of the image feature extraction model 1620 extract feature 4 (not shown in Figure 16) of the image to be processed 1601. F is less than the total number of feature extraction layers in the image feature extraction model 1620. F can correspond to the second number. N and F can be the same or different, which is not limited here.

[0251] After obtaining features 1-4, features 1 and 3 can be concatenated to obtain deep image features 1604 of the image to be processed 1601. Features 2 and 4 can be concatenated to obtain shallow image features 1605 of the image to be processed 1601.

[0252] The image 1601 to be processed is split into four image blocks. In Figure 16, only image block 1602 is marked. The dashed box 1603 includes four image blocks. The four image blocks included in the dashed box 1603 are input into the image feature extraction model 1610, and the image features 1 of the four image blocks can be obtained (not shown in Figure 16, only for the convenience of description). The image features 1 of the four image blocks can be stitched together to obtain the first fusion feature 1 (not shown in Figure 16).

[0253] The four image blocks included in the dashed box 1603 are all input into the image feature extraction model 1620, and the image features 2 of the four image blocks can be obtained. The image features 2 of the four image blocks can be spliced ​​to obtain the first fusion feature 2. Then, the first fusion feature 1 and the first fusion feature 2 can be fused to obtain the first fusion feature 1606 of the four image blocks.

[0254] The deep image features 1604 and the shallow image features 1605 can be input into the feature transformation sub-model 16302 of the feature processing model 1630. The feature transformation sub-model 16302 can perform attention calculation on the deep image features 1604 and the shallow image features 1605. For the features output by the feature transformation sub-model 16302, the dimension of the features output by the multilayer perceptron 16305 can be adjusted to obtain the first attention feature (not shown in Figure 16).

[0255] The deep image features 1604 and the first fusion features 1606 are input into the feature transformation sub-model 16301 of the feature processing model 1630. The feature transformation sub-model 16301 can perform attention calculation based on the deep image features 1604 and the first fusion features 1606. For the features output by the feature transformation sub-model 16301, the dimension of the features output by the multilayer perceptron 16304 can be adjusted to obtain the second attention features (not shown in Figure 16).

[0256] For the deep image feature 1604, the dimension of the deep image feature 1604 can be adjusted by the multilayer perceptron 16303. The dimension-adjusted deep image feature 1604 is then fused with the second attention feature to obtain the second fused feature 1607. The second fused feature 1607 can then be fused with the first attention feature to obtain the transformation feature 1608.

[0257] The dimension of the transformation feature 1608 is adjusted to the target dimension. In some embodiments, the vector parameters with the target dimension and the adjusted transformation feature 1608 can be input into the feature transformation sub-model 16306 to obtain the adjusted transformation feature. The prompt text can be input into the text feature extraction model 1650 to obtain the text features of the prompt text. Then, the text features and the adjusted transformation feature are input into the diffusion model 1640. The diffusion model 1640 can fuse the adjusted transformation feature and the text features to obtain the third fused feature. Random image noise is generated, and the image noise is iteratively denoised based on the third fused feature to generate the target image corresponding to the image to be processed 1601.

[0258] In the image generation method provided in this application embodiment, feature transformation can be performed on multi-scale image features with rich feature expression to obtain more accurate transformation features. The multi-scale image features with rich feature expression include richer detailed information and semantic information of the image to be processed. Correspondingly, the transformation features also include richer detailed information and semantic information of the image to be processed. Then, based on the transformation features and text features, a target image corresponding to the image to be processed can be generated. The target image can include richer detailed information of the image to be processed, and the semantics of the target image are consistent with the semantics of the prompt text, which can generate a more accurate target image. The diffusion model can improve the image quality of the generated target image. The image generation method provided in this application embodiment can improve the accuracy of generating the target image corresponding to the image to be processed and can improve the image quality of the generated target image.

[0259] The image generation method provided in this application embodiment is implemented through an image generation model, which consists of a diffusion model, an image feature extraction model, a text feature extraction model, and a feature processing model. The image feature extraction model can correspond to one or more of the first feature extraction model to the fourth feature extraction model mentioned above.

[0260] Referring to Figure 17, which is a schematic diagram of the structure of the image generation model provided in the embodiment of this application, the image feature extraction model 1701 is connected to the feature processing model 1702, the feature processing model 1702 is connected to the diffusion model 1704, and the text feature extraction model 1703 is connected to the diffusion model 1704.

[0261] In the image generation method provided in this application embodiment, the image feature extraction model and the feature processing model can constitute an extensible plug-in model. That is, users can increase the number of layers and dimensions of the feature processing model according to actual usage needs to improve the expressive power of the model, while maintaining the flexibility and extensibility of the model.

[0262] The feature processing model may include one or more feature transformation sub-models, which can be determined according to the scale of the image features. For example, in the case of multi-scale image features including deep image features, shallow image features, image features of multiple image patches, and image features of images to be processed at different resolutions, the number of feature transformation sub-models is at least four. These four feature transformation sub-models include: a feature transformation sub-model for performing attention calculations on deep and shallow image features; a feature transformation sub-model for performing attention calculations on deep image features and image features of multiple image patches; a feature transformation sub-model for performing attention calculations on deep image features and image features of images to be processed at different resolutions; and a feature transformation sub-model for adjusting the dimension of the transformed features to the target dimension. The feature transformation sub-model is a Transformer, which means that in this embodiment, a Transformer stacking structure can be used to add a feature transformation model and a feature extraction model to the diffusion model. For the feature transformation sub-model that adjusts the dimension of the transformed features to the target dimension, a noise addition step size can be input, thereby achieving time-series awareness.

[0263] The training method of the image generation model is explained below. The image generation model is trained in the following way: the sample image is input into the image feature extraction model to obtain the multi-scale image features of the sample image. The method of obtaining the multi-scale image features of the sample image can be referred to the description of the first to eighth implementation methods above, which will not be repeated here.

[0264] The multi-scale image features of the sample image are transformed using a feature processing model to obtain sample transformed features. See the explanation of step 102 in Figure 3 for details, which will not be repeated here. The sample prompt text of the sample image is input into a text feature extraction model to obtain sample text features. See the explanation of step 104 in Figure 3 for details, which will not be repeated here. Based on the sample transformed features and sample text features, a diffusion model is used to generate the predicted image corresponding to the sample image. See the explanation of step 104 in Figure 3 for details, which will not be repeated here.

[0265] Based on the model parameters of the diffusion model, image feature extraction model, and text feature extraction model in the image generation model, and the differences between the predicted image and the label images of the sample images, the model parameters of the feature processing model in the image generation model are adjusted. In other words, while keeping the model parameters of the diffusion model, image feature extraction model, and text feature extraction model unchanged, the model parameters of the feature processing model in the image generation model are adjusted based on the differences between the predicted image and the label images of the sample images.

[0266] In other words, by fixing the model parameters of the diffusion model, image feature extraction model, and text feature extraction model in the image generation model, and adjusting the model parameters of the feature processing model based on the difference between the predicted image and the label image of the sample image, an image generation model can be obtained. In this way, training of the image generation model can be achieved. This model is used to generate target images based on the image to be processed and the prompt text, improving the efficiency and accuracy of target image generation.

[0267] In some embodiments, there are multiple sample images, and the sample images are different in at least one of size and resolution. There is a correspondence between the sample images with different sizes and resolutions. For each sample image in the sample images with a correspondence, the sample image can be input into the image feature extraction model to obtain a sample image feature corresponding to a scale. That is, the multi-scale sample image features are obtained based on different sample images. The multi-scale sample image features are extracted separately from sample image branches of the same scene or the same object at different spatial resolutions.

[0268] In some embodiments, a corresponding label image exists for each corresponding sample image. The model parameters of the feature processing model in the image generation model can be adjusted based on the difference between the predicted image and the label image of the sample image, thereby obtaining the image generation model. This is equivalent to having multiple input images for the image generation model and outputting a single image.

[0269] In some embodiments, in order to further improve the accuracy of the image generation model, the label image of the sample image includes the target object, and the prediction image includes the prediction object. The model parameters of the feature processing model in the image generation model can be adjusted based on the model parameters of the diffusion model, the image feature extraction model, and the text feature extraction model in the image generation model, the difference between the label image of the prediction image and the sample image, and the difference between the prediction object and the target object.

[0270] In other words, by keeping the model parameters of the diffusion model, image feature extraction model, and text feature extraction model unchanged, the model parameters of the feature processing model in the image generation model are adjusted based on the differences between the label images of the predicted image and the sample images, as well as the differences between the predicted object and the target object. In this way, the model parameters of the feature processing model in the image generation model can be adjusted by combining the two differences, thereby improving the accuracy of the feature processing model, and thus improving the accuracy of the image generation model, so as to improve the accuracy and quality of the target image.

[0271] In some embodiments, the image generation model includes at least two training phases. In the first training phase, the sample image is determined as the label image of the sample image. Based on the model parameters of the diffusion model, image feature extraction model, and text feature extraction model in the image generation model, as well as the difference between the predicted image and the sample image, the model parameters of the feature processing model in the image generation model are adjusted.

[0272] In the second training phase, target images with semantic differences from sample images are identified as label images of sample images; based on the model parameters of the diffusion model, image feature extraction model, and text feature extraction model in the image generation model, as well as the differences between the predicted image and the target image, the model parameters of the feature processing model in the image generation model are adjusted.

[0273] In the first training phase, the sample image can be identified as the label image of the sample image, and the sample prompt text can be identified as the text describing the sample image. The model parameters of the diffusion model, the image feature extraction model, and the text feature extraction model can be kept unchanged. Based on the difference between the predicted image and the sample image, the model parameters of the feature processing model in the image generation model are adjusted.

[0274] In the second training phase, the target image with semantic differences from the sample image is identified as the label image of the sample image, and the sample prompt text is the text describing the target image. The model parameters of the diffusion model, image feature extraction model, and text feature extraction model can be kept unchanged. Based on the difference between the predicted image and the target image, the model parameters of the feature processing model in the image generation model are adjusted.

[0275] In some embodiments, there can be three training phases. The first training phase is consistent with the above, where the pixel resolution of the sample images is less than the pixel resolution threshold. In the first training phase, the sample images are unpaired samples. The second phase is consistent with the above, where the pixel resolution of the sample images is less than the pixel resolution threshold. In the second training phase, the sample images are paired samples.

[0276] The third training phase may include unpaired samples of the sample images whose label images are the target images, and paired samples whose label images are the target images whose pixel resolution is greater than a pixel resolution threshold.

[0277] To facilitate understanding, the three training stages are illustrated below. In some embodiments, the first stage uses unpaired samples for 512-resolution training, meaning the sample images have a pixel resolution of 512 by 512. These sample images are used to reconstruct the objects within the sample images while maintaining structural consistency. Compared to using sample images with a pixel resolution greater than or equal to a pixel resolution threshold in the first stage, using sample images with a pixel resolution less than the threshold improves the training efficiency of the first stage.

[0278] In the first training phase, by forcing the model to accurately reconstruct the original image based on the extracted image features, the feature processing model (i.e., the plug-in model) can quickly learn and align the object identity features and underlying texture details in the image to be processed, and establish a basic spatial and semantic mapping relationship.

[0279] The second training phase maintains a 512 resolution but switches to paired samples for training. Sample images are used as input, and the output is a target image whose semantics are consistent with the corresponding sample prompt text. Compared to the sample image, the target image differs from the sample image in at least one of the following: scene and object. The scene in the target image may differ from the sample image's scene. The object in the target image may differ from the object in the sample image in one or more of the following: action, pose, and style. Style refers to the image's style; for example, the image style could be cartoonish, realistic, cute, etc., and can be set according to actual usage requirements. In this way, the image generation model can generate target images with different actions, poses, and styles for objects given prompt text. The second training phase effectively eliminates the copy-paste effect between the target image and the image to be processed, enhancing text controllability.

[0280] In the second training phase, the paired samples can include image pairs of the same object from different perspectives, with different actions, or in different scenes. During this phase, the model learns how to decouple and recombine the bound object features with changing text prompts, thereby effectively eliminating the simple copy-paste effect and achieving true text-driven dynamic customization of objects.

[0281] The third training phase uses paired and unpaired samples with pixel resolutions greater than a pixel resolution threshold for joint training; that is, it uses high-resolution paired and unpaired samples for joint training. In this way, a limited number of high-resolution joint training sessions can significantly improve the visual quality and texture details of the generated target image.

[0282] The third training stage utilizes high-quality sample images to generate target images with fidelity exceeding a fidelity threshold (i.e., high fidelity) and controllable text. By training the image generation model in separate training stages, the quality of the images generated by the model can be improved at different stages. This facilitates the generation of target images that maintain consistency between the generated object and the image to be processed, and also improves the image quality of the generated target images. Furthermore, it enriches the training methods for the image generation model, thus fully training it and addressing technical problem 3.

[0283] In the third training phase, introducing high-resolution images activates the diffusion model's potential for generating high-frequency details. Simultaneously, joint fine-tuning using paired and unpaired samples prevents the model from losing the high-fidelity detail reconstruction capabilities learned in the first phase when pursuing text controllability, ensuring the final output is a high-fidelity, semantically varied, customized image.

[0284] The image generation method provided in this application embodiment can improve the consistency of objects included in the target image and the image to be processed by combining multi-scale image features. Furthermore, in the third training stage, sample images with pixel resolution greater than or equal to the pixel resolution threshold can be used for training, which can retain the detail generation capability of the diffusion model and improve the quality of the generated target image. In the process of training the image generation model, this application embodiment combines sample prompt text and sample images, which can enhance the controllability of the text for the generated target image.

[0285] Compared to the U-Net-based architecture in related technologies, the diffusion model exhibits unprecedented fidelity in generated images, providing a more robust foundation for image generation and editing tasks. However, related technologies, which combine reference images and text to generate target images based on the U-Net architecture, suffer from poor consistency between the objects included in the generated target image and the reference image, and the low fidelity of the generated target image limits its generalization ability in generating open-domain characters (i.e., objects in the image). Here, open-domain characters refer to a set of images encompassing cross-style (i.e., image style) categories such as anime, realism, and fantasy, and are not limited by a preset character library. The image generation method provided in this application, however, improves the consistency between the objects included in the generated target image and the image to be processed, enhances the fidelity of the generated target image, improves the generalization ability in generating open-domain characters (i.e., objects in the image), and achieves object customization (i.e., character customization), that is, customizing the scene in which the object is located and customizing the actions of the object, enabling the generation of diverse target images for the object.

[0286] In the image generation method provided in this application embodiment, object (i.e., character) driven image generation with strong versatility and high fidelity is achieved through an extended diffusion model. In this application embodiment, an extensible plug-in model consisting of an image feature extraction model and a feature processing model is provided. This model can effectively parse the features of objects in the image to be processed and seamlessly interact with the latent space of the diffusion model. Furthermore, in this application embodiment, multiple training stages exist when training the image generation model. These multiple training stages can adapt to diverse training samples, achieving phased training for object consistency and text editability.

[0287] For scalable plug-in models, traditional adapters based on the U-Net architecture, designed specifically for U-Net models and lacking scalability, cannot be directly applied to diffusion models. However, the plug-in model provided in this application uses a scalable full Transformer adapter for feature processing, serving as a crucial bridge between the conditional role image (i.e., the image to be processed) and the underlying model's latent generation space (i.e., the latent space of the diffusion model). The scalable plug-in model achieves scalability by increasing layer depth and hidden feature size.

[0288] In related technologies, although image feature extraction models can extract semantic information from images to be processed, they lack detailed texture information. In the image generation method provided in this application embodiment, SigLIP can be used as the main image feature extraction model and DINOv2 as the auxiliary image feature extraction model. SigLIP is good at capturing fine-grained object information in the image to be processed, while DINOv2 can reduce the interference of background factors in the image to be processed, thereby reducing feature loss and enhancing robustness. Multi-scale image features are extracted by SigLIP and DINOv2. The multi-scale image features with rich feature expression include richer detailed information and semantic information of the image to be processed.

[0289] In some embodiments, before extracting multi-scale image features using SigLIP and DINOv2, the image to be processed can be matted to obtain the objects in the image, or the transparency of the area outside the objects in the image to be processed can be set to 0 to obtain an image containing only the objects, and then multi-scale image features of the image containing only the objects can be extracted using SigLIP and DINOv2.

[0290] In some embodiments, SigLIP and DINOv2 may lead to the loss of fine-grained features when pre-trained with pixel resolution less than a pixel resolution threshold. The multi-scale image features extracted by SigLIP and DINOv2 in this application embodiment may include deep image features, shallow image features, and image features of multiple image blocks. Shallow image features can preserve details that are easily lost in deep layers. For image features of multiple image blocks, the image blocks are obtained by dividing the image to be processed. There are no overlapping areas between the image blocks. Each image block corresponds to a region of the image to be processed, which is equivalent to obtaining image features of image blocks at the region level.

[0291] Deep image features and shallow image features are transformed using a Transformer. Deep image features and image features from multiple image patches are also transformed using a Transformer to obtain transformed features. Essentially, deep image features, shallow image features, and image features from multiple image patches can be processed through two feature transformation paths. Each feature path is transformed by an independent Transformer, thus obtaining transformed features. By fusing the transformed features from different feature paths, a transformed feature containing multi-scale complementary information can be obtained by concatenating them along the token dimension.

[0292] A time-aware Transformer projects the transformation features of multi-scale complementary information into the denoising space of a diffusion model. These transformation features are treated as key-value pairs (i.e., as key vectors and value vectors), while a set of learnable query vectors (corresponding vector parameters) is dynamically updated through an attention mechanism. The dimensionality-adjusted transformation features are injected into the denoising space through a learnable cross-attention layer, achieving effective preservation of object identity features and flexible adaptation to complex text-driven modifications. In this embodiment, combining the image to be processed, including the object, and the prompt text, customized effects for the object can be generated.

[0293] The following continues to describe an exemplary structure of the image generation apparatus 455 provided in the embodiments of this application as a software module. In some embodiments, as shown in FIG2, the software module stored in the image generation apparatus 455 in the memory 450 may include:

[0294] Feature extraction module 4551 is configured to extract multi-scale image features of the image to be processed from the image to be processed;

[0295] Feature conversion module 4552 is configured to perform feature conversion on the multi-scale image features to obtain converted features;

[0296] The feature extraction module 4551 is further configured to extract features from the prompt text to obtain the text features of the prompt text;

[0297] The image generation module 4553 is configured to generate a target image corresponding to the image to be processed through a diffusion model based on the transformation features and the text features, wherein the semantics of the target image are consistent with the semantics of the prompt text.

[0298] In some embodiments, the feature extraction module 4551 is further configured to extract the deep image features of the image to be processed through M feature extraction layers in the first feature extraction model; extract the shallow image features of the image to be processed through N feature extraction layers in the first feature extraction model, wherein M is greater than N, and both M and N are integers greater than 0; and determine the deep image features and the shallow image features as the multi-scale image features of the image to be processed.

[0299] In some embodiments, the multi-scale image features further include mid-level image features; the feature extraction module 4551 is further configured to extract the mid-level image features of the image to be processed through P feature extraction layers in the first feature extraction model before determining the deep image features and the shallow image features as the multi-scale image features of the image to be processed, wherein P is greater than N and less than M;

[0300] The feature extraction module 4551 is further configured to determine the deep image features, the mid-level image features, and the shallow image features as multi-scale image features of the image to be processed.

[0301] In some embodiments, the feature conversion module 4552 is further configured to use the deep image features as a query vector and the shallow image features as a key vector and a value vector, respectively; and to perform attention calculation on the query vector, the key vector, and the value vector to obtain the converted features.

[0302] In some embodiments, the feature extraction module 4551 is further configured to divide the image to be processed into multiple image blocks of the same size before determining the deep image features and the shallow image features as multi-scale image features of the image to be processed; and for each image block, to extract features from the image block using the first feature extraction model to obtain the image features of the image block.

[0303] The feature extraction module 4551 is further configured to determine the deep image features, the shallow image features, and the image features of multiple image blocks as the multi-scale image features of the image to be processed.

[0304] In some embodiments, the feature conversion module 4552 is further configured to perform attention calculation on the deep image features and the shallow image features to obtain a first attention feature; perform attention calculation on the deep image features and the image features of the plurality of image blocks to obtain a second attention feature; and perform feature fusion on the first attention feature and the second attention feature to obtain the converted feature.

[0305] In some embodiments, the feature conversion module 4552 is further configured to fuse the image features of the plurality of image blocks to obtain a first fused feature; use the deep image features as a query vector, and use the first fused feature as a key vector and a value vector, respectively; and perform attention calculation on the query vector, the key vector, and the value vector to obtain a second attention feature.

[0306] In some embodiments, the feature conversion module 4552 is further configured to perform feature fusion with the second attention feature and the deep image feature to obtain a second fused feature; and to perform feature fusion with the second fused feature and the first attention feature to obtain a converted feature.

[0307] In some embodiments, the feature extraction module 4551 is further configured to adjust the resolution of the image block using the first feature extraction model to obtain an adjusted image block, wherein the resolution of the adjusted image block is adapted to the first feature extraction model; and to extract features from the adjusted image block using the first feature extraction model to obtain the image features of the adjusted image block.

[0308] The feature extraction module 4551 is further configured to determine the deep image features, the shallow image features, and the image features of the adjusted image blocks as the multi-scale image features of the image to be processed.

[0309] In some embodiments, the image to be processed includes a first image to be processed and a second image to be processed, wherein the pixel resolution of the first image to be processed is smaller than the pixel resolution of the second image to be processed;

[0310] The feature extraction module 4551 is further configured to extract image features of the first image to be processed through a second feature extraction model; extract image features of the second image to be processed through the second feature extraction model; and determine the image features of the first image to be processed and the image features of the second image to be processed as the multi-scale image features of the image to be processed.

[0311] The feature conversion module 4552 is further configured to perform attention calculation on the image features of the first image to be processed and the image features of the second image to be processed to obtain converted features.

[0312] In some embodiments, the image generation module 4553 is further configured to adjust the dimension of the transformation feature to the target dimension to obtain the adjusted transformation feature; and based on the adjusted transformation feature and the text feature, generate a target image corresponding to the image to be processed through a diffusion model.

[0313] In some embodiments, the image generation module 4553 is further configured to use a vector parameter of the target dimension as a query vector, and the transformation features as a key vector and a value vector, respectively; and to perform attention calculation on the query vector, the key vector, and the value vector to obtain the adjusted transformation features.

[0314] In some embodiments, the image to be processed includes an object and a background; the image generation module 4553 is further configured to fuse the adjusted transformation features and the text features through a diffusion model to obtain a third fusion feature; randomly generate image noise, and iteratively denoise the image noise based on the third fusion feature to generate a target image corresponding to the image to be processed, wherein the target image and the image to be processed differ from each other by at least one of the following:

[0315] The objects included in the target image are different from the objects included in the image to be processed;

[0316] The background included in the target image is different from the background included in the image to be processed.

[0317] In some embodiments, the feature extraction module 4551 is further configured to extract a first image feature of the image to be processed from the image to be processed through a third feature extraction model, the first image feature including multiple scales; extract a second image feature of the image to be processed from the image to be processed through a fourth feature extraction model, the second image feature including multiple scales; and fuse the first image feature and the second image feature to obtain multi-scale image features of the image to be processed.

[0318] In some embodiments, the image generation method is implemented through an image generation model, which consists of the diffusion model, the image feature extraction model, the text feature extraction model, and the feature processing model.

[0319] The feature extraction module 4551 is further configured to extract multi-scale image features of the image to be processed from the image to be processed through an image feature extraction model;

[0320] The feature conversion module 4552 is further configured to perform feature conversion on the multi-scale image features through the feature processing model to obtain converted features;

[0321] The feature extraction module 4551 is also configured to extract features from the prompt text using a text feature extraction model to obtain the text features of the prompt text.

[0322] In some embodiments, the software module stored in the image generation device 455 of the memory 450 may further include a model training module, wherein the image generation method is implemented by an image generation model, and the image generation model is composed of the diffusion model, the image feature extraction model, the text feature extraction model, and the feature processing model.

[0323] The model training module is configured to input sample images into the image feature extraction model to obtain multi-scale image features of the sample images; perform feature transformation on the multi-scale image features of the sample images through the feature processing model to obtain sample transformation features; input sample prompt text into the text feature extraction model to obtain sample text features; generate a predicted image corresponding to the sample image through the diffusion model based on the sample transformation features and the sample text features; and adjust the model parameters of the feature processing model in the image generation model based on the model parameters of the diffusion model, the image feature extraction model, and the text feature extraction model in the image generation model, as well as the difference between the predicted image and the label image of the sample image.

[0324] In some embodiments, the image generation model includes at least two training phases; the model training module is further configured to, in the first training phase, determine the sample image as the label image of the sample image; adjust the model parameters of the feature processing model in the image generation model based on the model parameters of the diffusion model, the image feature extraction model, and the text feature extraction model in the image generation model, and the difference between the predicted image and the sample image; in the second training phase, determine the target image with semantically different from the sample image as the label image of the sample image; adjust the model parameters of the feature processing model in the image generation model based on the model parameters of the diffusion model, the image feature extraction model, and the text feature extraction model in the image generation model, and the difference between the predicted image and the target image.

[0325] In some embodiments, the label image of the sample image includes a target object, and the prediction image includes a prediction object; the model training module is further configured to adjust the model parameters of the feature processing model in the image generation model based on the model parameters of the diffusion model, the image feature extraction model, and the text feature extraction model in the image generation model, the difference between the label image of the prediction image and the sample image, and the difference between the prediction object and the target object.

[0326] In some embodiments, the number of sample images is multiple, and the size and resolution of the multiple sample images are different; the model training module is further configured to input the sample image into the image feature extraction model for each sample image to obtain a sample image feature corresponding to a scale.

[0327] In some embodiments, the image to be processed includes an object and a background; the feature extraction module 4551 is further configured to perform image segmentation on the image to be processed, determine the object and the background in the image to be processed; set the transparency of the segmented background to 0 to obtain an image to be processed with adjusted transparency; and extract multi-scale image features from the image to be processed with adjusted transparency.

[0328] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the image generation method described in this application.

[0329] This application provides a computer-readable storage medium storing computer-executable instructions or computer programs. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the image generation method provided in this application, such as the image generation method shown in FIG3.

[0330] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0331] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0332] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0333] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0334] In summary, the image processing method provided in this application embodiment can perform feature transformation on multi-scale image features with rich feature expression, resulting in more accurate transformed features. These richly expressed multi-scale image features include richer detailed and semantic information of the image to be processed. Correspondingly, the transformed features also include richer detailed and semantic information of the image to be processed. Based on the transformed features and text features, a target image corresponding to the image to be processed can be generated. The target image includes richer detailed information of the image to be processed, and its semantics are consistent with the semantics of the prompt text, resulting in a more accurate target image. The diffusion model improves the image quality of the generated target image. The image generation method provided in this application embodiment can improve the accuracy of generating target images corresponding to the image to be processed and improve the image quality of the generated target images, thus solving technical problems 1 and 2. The image generation scheme provided in this application embodiment can achieve object customization, that is, customize the scene in which the object is located and customize the action of the object, enabling the generation of diverse target images for the object.

[0335] By training the image generation model in stages, the quality of the images generated by the model can be improved at different stages. This makes it easier to generate target images that maintain consistency between the object and the image to be processed, and improves the image quality of the generated target images. It also enriches the training methods of the image generation model, thereby fully training the image generation model and solving technical problem 3.

[0336] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An image generation method, the method being performed by an electronic device, the method comprising: Extract multi-scale image features from the image to be processed; The multi-scale image features are transformed to obtain transformed features; Feature extraction is performed on the prompt text to obtain its text features; Based on the transformation features and the text features, a target image corresponding to the image to be processed is generated using a diffusion model; The semantics of the target image are consistent with the semantics of the prompt text.

2. The method of claim 1, wherein, The multi-scale image features include deep image features and shallow image features; The extraction of multi-scale image features from the image to be processed includes: The deep image features of the image to be processed are extracted through M feature extraction layers in the first feature extraction model. The shallow image features of the image to be processed are extracted through N feature extraction layers in the first feature extraction model, where M is greater than N, and both M and N are integers greater than 0. The deep image features and the shallow image features are determined as the multi-scale image features of the image to be processed.

3. The method of claim 2, wherein, The multi-scale image features further include mid-level image features; before determining the deep image features and the shallow image features as the multi-scale image features of the image to be processed, the method further includes: The mid-level image features of the image to be processed are extracted through P feature extraction layers in the first feature extraction model, wherein P is greater than N and less than M; The step of determining the deep image features and the shallow image features as the multi-scale image features of the image to be processed includes: The deep image features, the mid-level image features, and the shallow image features are determined as the multi-scale image features of the image to be processed.

4. The method of claim 2, wherein, The step of performing feature transformation on the multi-scale image features to obtain transformed features includes: The deep image features are used as the query vector, and the shallow image features are used as the key vector and value vector, respectively. Attention is calculated on the query vector, the key vector, and the value vector to obtain the transformation features.

5. The method of claim 2, wherein, Before determining the deep image features and the shallow image features as the multi-scale image features of the image to be processed, the method further includes: The image to be processed is divided into multiple image blocks of the same size; For each image block, features are extracted from the image block using the first feature extraction model to obtain the image features of the image block; The step of determining the deep image features and the shallow image features as the multi-scale image features of the image to be processed includes: The deep image features, the shallow image features, and the image features of the multiple image blocks are determined as the multi-scale image features of the image to be processed.

6. The method of claim 5, wherein, The step of performing feature transformation on the multi-scale image features to obtain transformed features includes: Attention is calculated on the deep image features and the shallow image features to obtain the first attention feature; Attention calculations are performed on the deep image features and the image features of multiple image blocks to obtain the second attention features; The first attention feature and the second attention feature are fused to obtain the transformation feature.

7. The method of claim 6, wherein, The step of fusing the first attention feature and the second attention feature to obtain the transformation feature includes: The image features of multiple image blocks are fused to obtain a first fused feature; The deep image features are used as the query vector, and the first fused features are used as the key vector and the value vector, respectively. Attention is calculated on the query vector, the key vector, and the value vector to obtain the second attention feature.

8. The method of claim 6, wherein, The first attention feature and the second attention feature are then fused. Obtaining the transformation features includes: The second attention feature is fused with the deep image feature to obtain the second fused feature; The second fusion feature is fused with the first attention feature to obtain the transformation feature.

9. The method according to any one of claims 5-8, wherein, The step of extracting features from the image patch using the first feature extraction model to obtain the image features of the image patch includes: The resolution of the image patch is adjusted using the first feature extraction model to obtain an adjusted image patch, wherein the resolution of the adjusted image patch is adapted to the first feature extraction model. The first feature extraction model is used to extract features from the adjusted image patch to obtain the image features of the adjusted image patch; The step of determining the deep image features and the shallow image features as the multi-scale image features of the image to be processed includes: The deep image features, the shallow image features, and the image features of the adjusted image blocks are determined as the multi-scale image features of the image to be processed.

10. The method of claim 1, wherein, The image to be processed includes a first image to be processed and a second image to be processed, wherein the pixel resolution of the first image to be processed is smaller than the pixel resolution of the second image to be processed; The extraction of multi-scale image features from the image to be processed includes: The image features of the first image to be processed are extracted using the second feature extraction model. The second feature extraction model is used to extract image features from the second image to be processed. The image features of the first image to be processed and the image features of the second image to be processed are determined as the multi-scale image features of the image to be processed; The step of performing feature transformation on the multi-scale image features to obtain transformed features includes: Attention calculations are performed on the image features of the first image to be processed and the image features of the second image to be processed to obtain the transformation features.

11. The method of any one of claims 1-10, wherein, The step of generating the target image corresponding to the image to be processed through a diffusion model based on the transformation features and the text features includes: Adjust the dimension of the transformation feature to the target dimension to obtain the adjusted transformation feature; Based on the adjusted transformation features and the text features, a target image corresponding to the image to be processed is generated using a diffusion model.

12. The method of claim 11, wherein, The process of adjusting the dimension of the transformed feature to the target dimension to obtain the adjusted transformed feature includes: The vector parameter with the dimension of the target dimension is used as the query vector, and the transformation features are used as the key vector and value vector, respectively. Attention is calculated on the query vector, the key vector, and the value vector to obtain the adjusted transformation features.

13. The method of claim 10 or 11, wherein, The image to be processed includes an object and a background; the step of generating a target image corresponding to the image to be processed based on the adjusted transformation features and the text features using a diffusion model includes: The adjusted transformation features and the text features are fused using a diffusion model to obtain a third fused feature; Randomly generate image noise, and iteratively denoise the image noise based on the third fusion feature to generate a target image corresponding to the image to be processed, wherein the target image and the image to be processed differ from each other by at least one of the following: The objects included in the target image are different from the objects included in the image to be processed; The background included in the target image is different from the background included in the image to be processed.

14. The method of claim 1, wherein, The extraction of multi-scale image features from the image to be processed includes: A third feature extraction model is used to extract a first image feature from the image to be processed, the first image feature including multiple scales; The second image feature of the image to be processed is extracted from the image to be processed using the fourth feature extraction model. The second image feature includes multiple scales. By fusing the first image features and the second image features, multi-scale image features of the image to be processed are obtained.

15. The method of any one of claims 1-14, wherein, The image generation method is implemented through an image generation model, which consists of a diffusion model, an image feature extraction model, a text feature extraction model, and a feature processing model. The image generation model is trained in the following way: The sample image is input into the image feature extraction model to obtain the multi-scale image features of the sample image; The multi-scale image features of the sample image are transformed using the feature processing model to obtain the sample transformed features. The sample prompt text is input into the text feature extraction model to obtain the sample text features; Based on the sample transformation features and the sample text features, a predicted image corresponding to the sample image is generated using the diffusion model; Based on the model parameters of the diffusion model, the image feature extraction model, and the text feature extraction model in the image generation model, and the difference between the predicted image and the label image of the sample image, the model parameters of the feature processing model in the image generation model are adjusted.

16. The method of claim 15, wherein, The image generation model includes at least two training phases; the adjustment of the model parameters of the feature processing model in the image generation model based on the model parameters of the diffusion model, the image feature extraction model, and the text feature extraction model in the image generation model, and the difference between the predicted image and the label image of the sample image, includes: In the first training phase, the sample image is determined as the label image of the sample image; based on the model parameters of the diffusion model, the image feature extraction model and the text feature extraction model in the image generation model, and the difference between the predicted image and the sample image, the model parameters of the feature processing model in the image generation model are adjusted; In the second training phase, a target image whose semantics differs from the sample image is identified as the label image of the sample image; based on the model parameters of the diffusion model, the image feature extraction model, and the text feature extraction model in the image generation model, and the difference between the predicted image and the target image, the model parameters of the feature processing model in the image generation model are adjusted.

17. The method of claim 15, wherein, The label image of the sample image includes a target object, and the predicted image includes a predicted object; the adjustment of the model parameters of the feature processing model in the image generation model based on the model parameters of the diffusion model, the image feature extraction model, and the text feature extraction model in the image generation model, and the difference between the predicted image and the label image of the sample image, includes: Based on the model parameters of the diffusion model, the image feature extraction model, and the text feature extraction model in the image generation model, the difference between the label image of the predicted image and the sample image, and the difference between the predicted object and the target object, the model parameters of the feature processing model in the image generation model are adjusted.

18. The method of claim 1, wherein, The image to be processed includes an object and a background; the extraction of multi-scale image features from the image to be processed includes: The image to be processed is segmented to determine the object and the background in the image to be processed; The transparency of the segmented background is set to 0 to obtain the image to be processed after transparency adjustment. Multi-scale image features are extracted from the image to be processed after transparency adjustment.

19. An image generation apparatus, the apparatus being disposed in an electronic device, the apparatus comprising: The feature extraction module is configured to extract multi-scale image features from the image to be processed; The feature conversion module is configured to perform feature conversion on the multi-scale image features to obtain converted features; The feature extraction module is further configured to extract features from the prompt text to obtain the text features of the prompt text; The image generation module is configured to generate a target image corresponding to the image to be processed through a diffusion model based on the transformation features and the text features, wherein the semantics of the target image are consistent with the semantics of the prompt text.

20. An electronic device, the electronic device comprising: Memory is used to store executable instructions or computer programs. A processor, configured to execute computer-executable instructions or computer programs stored in the memory, implements the image generation method according to any one of claims 1 to 18.

21. A computer readable storage medium storing computer executable instructions or a computer program which, when executed by a processor, implement the image generation method of any one of claims 1 to 18.

22. A computer program product comprising computer executable instructions or a computer program which, when executed by a processor, implement the image generation method of any one of claims 1 to 18.

Images