Image generation method and apparatus, and device, medium and program product
By determining the style of the target image and selecting the corresponding model, a new image is generated using self-attention block stitching features. This solves the problem of inconsistent styles in new images in existing technologies and achieves a high degree of similarity between the new image and the original image.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- SHANGHAI SOULGATE TECH CO LTD
- Filing Date
- 2025-07-31
- Publication Date
- 2026-06-04
AI Technical Summary
Existing technologies struggle to generate new images that match the style of the original image, making it impossible for the new image to completely replace the original.
By acquiring the subject and textual description information of the target image, the image style is determined, and an appropriate image generation model is selected. New images are generated using modules for segmentation, encoding, random generation, noise addition, noise reduction, and decoding. By combining self-attention blocks to stitch together the subject and random noise features, new images with consistent styles are generated.
The generated new image has a subject and style that are highly consistent with the original image, and can effectively replace the original image.
Smart Images
Figure CN2025111813_04062026_PF_FP_ABST
Abstract
Description
An image generation method, apparatus, device, medium, and program product
[0001] This application claims priority to Chinese Patent Application No. 202411732172.8, filed on November 28, 2024, entitled "An Image Generation Method, Apparatus, Device, Medium and Program Product", the entire contents of which are incorporated herein by reference. Technical Field
[0002] This application relates to the field of computer technology, and in particular to an image generation method, apparatus, device, medium, and program product. Background Technology
[0003] Currently, based on image generation technology, a trained image generation model can be used to generate new images with the same subject as a given image, as shown in Figure 1. Given image A, new images A1 to A4 can be obtained, all featuring dogs as the subject. However, the style of these new images is not consistent with that of image A; that is, the new images are only similar to the original images and cannot completely replace them.
[0004] Therefore, how to generate a new image that can replace the original image is a problem that needs to be solved by those skilled in the art. Summary of the Invention
[0005] In view of this, the purpose of this application is to provide an image generation method, apparatus, device, medium, and program product to generate a new image that can replace the original image. The specific solution is as follows:
[0006] In a first aspect, this application provides an image generation method, including:
[0007] Acquire a target image including the target subject and textual description information of the target image, and determine the image style of the target image;
[0008] Select the target model corresponding to the image style from a set of preset image generation models;
[0009] The target image is input into the target model, so that the target model splices the main image features of the target subject and the random image features of random noise, and generates a new image based on the splicing result, the main image mask of the target subject and the text description information.
[0010] Optionally, determining the image style of the target image includes:
[0011] The target image is compared with a preset image library to identify known style images in the image library that have a similarity greater than a preset threshold to the target image.
[0012] The image style of the known style image is used as the image style of the target image;
[0013] or
[0014] The target images are clustered, and the image style of the target images is determined based on the clustering results;
[0015] or
[0016] The target image is input into a preset style classification model so that the style classification model outputs the image style of the target image.
[0017] Optionally, a target model corresponding to the image style is selected from a plurality of preset image generation models, including:
[0018] Determine the style identifier information of the image style;
[0019] The image generation model marked with the style identification information among the multiple image generation models is used as the target model.
[0020] Optionally, the target model includes: a segmentation module, an encoding module, a random generation module, a noise-adding module, a noise-reducing module, and a decoding module;
[0021] The segmentation module is used to extract the subject mask from the target image;
[0022] The encoding module is used to encode the target image into latent space image features;
[0023] The random generation module is used to generate the random noise at any time;
[0024] The noise-adding module is used to add noise to the latent space image features to obtain noisy image features;
[0025] The denoising module is used to splice the main image features and the random image features, and generate new image features based on the splicing result, the main image mask and the text description information;
[0026] The decoding module is used to decode the new image features into the new image.
[0027] Optionally, the denoising module includes a self-attention block;
[0028] The self-attention block is used to splice the main image features and the random image features, and to generate new image features based on the splicing result, the main mask, and the text description information.
[0029] Optionally, the self-attention block is used to splice the main image features and the random image features using the first formula and the second formula;
[0030] The first formula is: K cat =concatenate(K,K) ref The second formula is: V cat =concatenate(V,V) ref The main image features include the main key matrix K. ref and principal value matrix V ref The random image features include a random key matrix K and a random value matrix V; the concatenated result includes a concatenation key matrix K. cat and spliced value matrix V cat .
[0031] Optionally, the self-attention block is used to combine the result obtained by fusing and splicing the third, fourth, and fifth formulas with the main body mask;
[0032] The third formula is: M cat =concatenate(1,M) ref The fourth formula is: The fifth formula is: H = ATT * V cat ;
[0033] Among them, M cat This represents the concatenation result of the random noise and the main body mask; M ref The main body mask is represented by ATT; the self-attention weights are represented by H; and the concatenation matrix V is represented by H. cat The weighted result of the self-attention weight value ATT; Q represents the query matrix of the target subject; Represents the splicing key matrix K cat The transpose of .
[0034] Secondly, this application provides an image generation apparatus, comprising:
[0035] The acquisition module is used to acquire a target image including the target subject and text description information of the target image, and to determine the image style of the target image;
[0036] The selection module is used to select the target model corresponding to the image style from a plurality of preset image generation models;
[0037] The generation module is used to input the target image into the target model, so that the target model splices the main image features of the target subject and the random image features of random noise, and generates a new image based on the splicing result, the main mask of the target subject and the text description information.
[0038] Thirdly, this application provides an electronic device, comprising:
[0039] Memory, used to store computer programs;
[0040] A processor is configured to execute the computer program to implement the aforementioned disclosed image generation method.
[0041] Fourthly, this application provides a readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned disclosed image generation method.
[0042] Fifthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the aforementioned disclosed image generation method.
[0043] As can be seen from the above scheme, this application provides an image generation method, including: acquiring a target image including a target subject and text description information of the target image, and determining the image style of the target image; selecting a target model corresponding to the image style from a plurality of preset image generation models; inputting the target image into the target model, so that the target model splices the subject image features of the target subject and the random image features of random noise, and generates a new image based on the splicing result, the subject mask of the target subject and the text description information.
[0044] As can be seen, this application can select a target image generation model that corresponds to the image style of the target image based on the given target image. Then, the target image generation model can integrate the subject image, subject mask and text description information of the target image to generate a new image, so that the subject and style of the new image are consistent with the subject and style of the given original target image, and the two are more similar, so that the new image can replace the original image.
[0045] Correspondingly, the image generation apparatus, device, medium, and program product provided in this application also have the above-mentioned technical effects. Attached Figure Description
[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0047] Figure 1 is a schematic diagram of an image generation method disclosed in this application;
[0048] Figure 2 is a flowchart of an image generation method disclosed in this application;
[0049] Figure 3 is a schematic diagram of the structure of a Unet disclosed in this application;
[0050] Figure 4 is a comparison of the image generation effects of different image generation schemes disclosed in this application;
[0051] Figure 5 is a schematic diagram of an electronic device disclosed in this application. Detailed Implementation
[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0053] Currently, image generation technology can generate new images with the same subject as a given image, as shown in Figure 1. Given image A, new images A1 to A4 can be obtained, all featuring dogs as the subject. However, the style of these new images is not consistent with that of image A; that is, the new images are only similar to the original images and cannot completely replace them. Therefore, this application provides an image generation scheme that can generate new images with the same subject and style as a given original target image, thus replacing the original image.
[0054] Referring to Figure 2, this application discloses an image generation method, including:
[0055] S201. Obtain the target image including the target subject and the text description information of the target image, and determine the image style of the target image.
[0056] In one embodiment, determining the image style of a target image includes: comparing the target image with a preset image library to identify known style images in the image library whose similarity to the target image is greater than a preset threshold; using the image style of the known style images as the image style of the target image; or clustering the target images and determining the image style of the target image based on the clustering results; or inputting the target image into a preset style classification model so that the style classification model outputs the image style of the target image.
[0057] S202. Select the target model corresponding to the image style from multiple preset image generation models.
[0058] This embodiment has multiple preset image generation models, each corresponding to a different image style. Image styles can include: traditional Chinese style, science fiction, 3D realistic, 3D cartoon, and anime / cartoon styles, etc.
[0059] In one implementation, selecting a target model corresponding to an image style from a plurality of preset image generation models includes: determining style identifier information of the image style; and using the image generation model marked with style identifier information from the plurality of image generation models as the target model.
[0060] S203. Input the target image into the target model so that the target model splices the main image features of the target subject and the random image features of random noise, and generates a new image based on the splicing result, the main mask of the target subject and the text description information.
[0061] In one implementation, the target model includes: a segmentation module, an encoding module, a random generation module, a noise-adding module, a noise-reducing module, and a decoding module; the segmentation module is used to extract a subject mask from the target image; the encoding module is used to encode the target image into latent space image features; the random generation module is used to generate random noise at any time; the noise-adding module is used to add noise to the latent space image features to obtain noisy image features; the noise-reducing module is used to splice the subject image features and random image features, and generate new image features based on the spliced result, the subject mask, and text description information; the decoding module is used to decode the new image features into a new image.
[0062] In one implementation, the denoising module includes a self-attention block; the self-attention block is used to stitch together the main image features and random image features, and generate new image features based on the stitched result, the main mask, and text description information.
[0063] Specifically, the self-attention block is used to concatenate the main image features and random image features using the first formula and the second formula; wherein, the first formula is: K cat =concatenate(K,K) ref The second formula is: Vcat =concatenate(V,V) ref The main image features include the main key matrix K. ref and principal value matrix V ref The random image features include a random key matrix K and a random value matrix V; the concatenated result includes a concatenation key matrix K. cat and spliced value matrix V cat .
[0064] In one implementation, the self-attention block is used to fuse and splice the result obtained by the third, fourth, and fifth formulas with the main body mask; wherein, the third formula is: M cat =concatenate(1,M) ref The fourth formula is: The fifth formula is: H = ATT * V cat Among them, M cat This represents the concatenation result of the random noise and the main body mask; M ref The main body mask is represented by ATT; the self-attention weights are represented by H; and the concatenation matrix V is represented by H. cat The weighted result of the self-attention weight value ATT; Q represents the query matrix of the target subject; Represents the splicing key matrix K cat The transpose of .
[0065] In this embodiment, Q is the query matrix, K is the key matrix, and V is the value matrix. K is used to calculate the association matrix ATT between Q and V.
[0066] It should be noted that a single image generation model often struggles to learn every style effectively. Therefore, this application pre-trains multiple image generation models with different styles. Given a reference image, one of these models is selected to learn the corresponding style for generating a new image. The image style of the reference image can be determined through clustering, classification, or other methods. After determining the image style of the reference image, it is input into the corresponding style's image generation model. The encoder in the image generation model transforms the reference image into a reference latent (undenoised) in the latent space. Then, the forward denoising module in the model adds noise to the reference latent, resulting in denoised reference latents at each time step. A noisy latent is randomly initialized, and both the noisy latent and the denoised reference latent are fed into the Unet (denoising module) in the image generation model. The denoising results are output after passing through each block of the Unet. In this process, when the noisy latent and the noisy reference latent pass through the self-attention blocks in each block of UNet, the self-attention blocks concatenate the random latent with the reference latent at the current time step using key-value pairs (K and V). This means the self-attention blocks adaptively learn information from the reference image to ensure consistency of the subject image. Simultaneously, a mask of the reference subject is introduced, and self-attention is calculated again to inject reference subject information into the new image. After denoising once by UNet, the output of UNet is input back into UNet, and the above denoising process is repeated until the denoised latent at the last time step is obtained. The denoised latent is then decoded by the decoder in the image generation model to obtain a new image with a style consistent with the subject. In other words, after the final denoising, the final denoising result is decoded to obtain the newly generated image.
[0067] In the forward noise addition module, the noise addition result at each step is retained during the process of adding noise to the latent portion of the reference image.
[0068] Furthermore, the self-attention block extracts the matrices q, k, v and q_ref, k_ref, v_ref of the random latent and the reference latent respectively. Then, it concatenates the k_ref, v_ref of the reference latent and the k, v of the random latent together before performing self-attention calculation. The purpose of this is to incorporate information from the reference image into the image generation process, thereby ensuring that the generated image is consistent with the reference image.
[0069] Please refer to Figure 3. Unet consists of one Conv_in, three CrossAttnDownBlock2D, one DownBlock2D, one UnetMidBlock2DCcrossAttn, one UpBlock2D, three CrossAttnUpBlock2D, and one Conv_out. Each module containing CrossAttn includes a residual block (fused with time-step features), a self-attention block (performing its own feature extraction), and a cross-attention block (fused with the input text information for feature fusion). The self-attention block mainly performs K and V concatenation of the random latent and the reference latent.
[0070] Specifically, the processing flow of the random latent and reference latent in UNet includes: In the first time step, the input random latent sequentially passes through a Conv_in, three CrossAttnDownBlock2D blocks, one DownBlock2D block, one UnetMidBlock2DCcrossAttn block, one UpBlock2D block, three CrossAttnUpBlock2D blocks, and a Conv_out block to obtain the predicted noise. When passing through the self-attention block in CrossAttn, K and V concatenation of the input random latent and reference latent is performed, and a mask of the reference subject is introduced for self-attention calculation, thereby injecting reference subject information. Based on the noise predicted by UNet, the denoised latent is obtained. The denoised latent is then fed back into UNet, and the above process is repeated for denoising until the denoised latent of the last time step is obtained.
[0071] In one example, the concatenation of K and V values between the random latent and the reference latent, as well as the injection calculation of the reference subject information, can be performed using the following formula:
[0072] K cat =concatenate(K,K) ref ), used to join two Ks;
[0073] V cat =concatenate(V,V) ref ), used to join two Vs;
[0074] M cat =concatenate(1,M) ref ), used to stitch together the main Mask and random latent of the reference image;
[0075] H = ATT * V cat ;
[0076] Among them, K ref V ref M was calculated from the reference latent respectively. ref Then it is the mask of the reference image, ATT is the self-attention weight added to the mask, and H is the output result after ATT weighting.
[0077] Please refer to Figure 4. By comparing the images generated by this application with those generated by other solutions, it can be seen that the new images generated by this application are closer to the reference image, specifically in terms of the subject and style of the person being closer to the reference image.
[0078] The following describes an image generation apparatus provided by an embodiment of this application. The image generation apparatus described below and the image generation method described above can be referred to each other.
[0079] This application discloses an image generation apparatus, including:
[0080] The acquisition module is used to acquire the target image, including the target subject and the text description information of the target image, and to determine the image style of the target image;
[0081] The selection module is used to select the target model corresponding to the image style from a number of preset image generation models;
[0082] The generation module is used to input the target image into the target model so that the target model can stitch together the main image features of the target subject and the random image features of random noise, and generate a new image based on the stitched result, the main image mask of the target subject and text description information.
[0083] In one implementation, the acquisition module is specifically used for:
[0084] The target image is compared with a preset image library to identify known style images in the image library that have a similarity greater than a preset threshold to the target image.
[0085] Use the image style of a known style image as the image style of the target image;
[0086] or
[0087] Cluster the target images and determine the image style of the target images based on the clustering results;
[0088] or
[0089] The target image is input into a preset style classification model so that the style classification model outputs the image style of the target image.
[0090] In one implementation, the selection module is specifically used for:
[0091] Determine the style identifier information for the image style;
[0092] The image generation model with style identifier information among multiple image generation models is used as the target model.
[0093] In one implementation, the target model includes: a segmentation module, an encoding module, a random generation module, a noise-adding module, a noise-reducing module, and a decoding module;
[0094] The segmentation module is used to extract the subject mask from the target image;
[0095] The encoding module is used to encode the target image into latent space image features;
[0096] The random generation module is used to generate random noise at any time;
[0097] The noise-adding module is used to add noise to the latent space image features to obtain noisy image features;
[0098] The denoising module is used to stitch together the main image features and random image features, and to generate new image features based on the stitched result, the main image mask, and the text description information.
[0099] The decoding module is used to decode new image features into a new image.
[0100] In one implementation, the noise reduction module includes a self-attention block;
[0101] Self-attention blocks are used to stitch together main image features and random image features, and generate new image features based on the stitched result, the main image mask, and text description information.
[0102] In one implementation, the self-attention block is used to splice main image features and random image features using a first formula and a second formula; wherein, the first formula is: K cat =concatenate(K,K) ref The second formula is: V cat =concatenate(V,V) ref The main image features include the main key matrix K. ref and principal value matrix V ref The random image features include a random key matrix K and a random value matrix V; the concatenated result includes a concatenation key matrix K. cat and spliced value matrix V cat .
[0103] In one implementation, the self-attention block is used to fuse and splice the result obtained by the third, fourth, and fifth formulas with the main body mask; wherein, the third formula is: M cat =concatenate(1,M) ref The fourth formula is: The fifth formula is: H = ATT * V cat Among them, M cat This represents the concatenation result of the random noise and the main body mask; M ref The main body mask is represented by ATT; the self-attention weights are represented by H; and the concatenation matrix V is represented by H. cat The weighted result of the self-attention weight value ATT; Q represents the query matrix of the target subject; Represents the splicing key matrix K cat The transpose of .
[0104] For more detailed information on the working process of each module and unit in this embodiment, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.
[0105] As can be seen, this embodiment provides an image generation apparatus that can generate a new image that can replace the original image, with a subject and style consistent with the subject and style of the given original target image.
[0106] The following describes an electronic device provided by an embodiment of this application. The electronic device described below can be referred to in conjunction with the image generation method and apparatus described above.
[0107] Referring to Figure 5, an embodiment of this application discloses an electronic device, including:
[0108] Memory 501 is used to store computer programs;
[0109] Processor 502 is configured to execute the computer program to implement the method disclosed in any of the above embodiments.
[0110] The following describes a readable storage medium provided in an embodiment of this application. The readable storage medium described below can be referred to in conjunction with the image generation method, apparatus and device described above.
[0111] A readable storage medium is provided for storing a computer program, wherein the computer program, when executed by a processor, implements the image generation method disclosed in the foregoing embodiments. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.
[0112] The following describes a computer program product provided by an embodiment of this application. The computer program product described below can be referred to in conjunction with other embodiments described herein.
[0113] A computer program product includes a computer program / instructions that, when executed by a processor, implement the steps of the aforementioned disclosed image generation method.
[0114] The terms “first,” “second,” “third,” “fourth,” etc., used in this application (if applicable) are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, or apparatus that includes a series of steps or units is not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, or apparatus.
[0115] It should be noted that the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed in this application.
[0116] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0117] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of readable storage medium known in the art.
[0118] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. An image generation method characterized by, The method comprises the following steps: obtaining a target image including a target subject and text description information of the target image, and determining an image style of the target image; selecting a target model corresponding to the image style from a plurality of preset image generation models; inputting the target image into the target model, so that the target model splices subject image features of the target subject and random image features of random noise, and generates a new image based on a spliced result, a subject mask of the target subject and the text description information.
2. The method of claim 1, wherein, Determining the image style of the target image comprises: comparing the target image with a preset image library to determine a known style image with a similarity greater than a preset threshold in the image library; taking the image style of the known style image as the image style of the target image; or clustering the target image and determining the image style of the target image according to a clustering result; or inputting the target image into a preset style classification model to output the image style of the target image.
3. The method of claim 1, wherein, Selecting a target model corresponding to the image style from a plurality of preset image generation models comprises: determining style identification information of the image style; taking an image generation model marked with the style identification information in the plurality of image generation models as the target model.
4. The method of claim 1, wherein, The target model comprises a segmentation module, an encoding module, a random generation module, a noise adding module, a noise removing module and a decoding module; the segmentation module is used for extracting the subject mask from the target image; the encoding module is used for encoding the target image into an implicit space image feature; the random generation module is used for generating the random noise at any time; the noise adding module is used for adding noise to the implicit space image feature to obtain a noisy image feature; the noise removing module is used for splicing the subject image features and the random image features, and generating a new image feature based on a spliced result, the subject mask and the text description information; the decoding module is used for decoding the new image feature into the new image.
5. The method of claim 4, wherein, The noise removing module comprises a self-attention block; the self-attention block is used for splicing the subject image features and the random image features, and generating a new image feature based on a spliced result, the subject mask and the text description information.
6. The method of claim 5, wherein, The self-attention block is used for splicing the subject image features and the random image features by using a first formula and a second formula; The first formula is K cat = concatenate(K, K ref ); the second formula is V cat = concatenate(V, V ref ); the subject image feature includes a subject key matrix K ref and a subject value matrix V ref ; the random image feature includes a random key matrix K and a random value matrix V; and the result of concatenation includes a concatenated key matrix K cat and a concatenated value matrix V cat .
7. The method of claim 5, wherein, the self-attention block is used for fusing the spliced result and the subject mask by using a third formula, a fourth formula and a fifth formula. wherein the third formula is: M cat = concatenate(1, M ref ); and the fourth formula is: The fifth formula is: H = ATT*V cat ; wherein M cat represents the splicing result of the random noise and the subject mask; M ref represents the subject mask; ATT represents a self-attention weight value; H represents a splicing value matrix V cat and the weighted result of the self-attention weight value ATT; Q represents a query matrix of the target subject; represents the transpose of the concatenation key matrix K cat of the concatenation key matrix K.
8. An image generation apparatus characterized by comprising: The method comprises the following steps: an obtaining module is configured to obtain a target image including a target subject and text description information of the target image, and determine an image style of the target image; a selecting module is configured to select a target model corresponding to the image style from a plurality of preset image generation models; The generating module is configured to input the target image into the target model, so that the target model splices the subject image feature of the target subject and the random image feature of the random noise, and generates a new image based on a result of the splicing, a subject mask of the target subject, and the text description information.
9. An electronic device, comprising: The computer program product comprises: a memory for storing a computer program; a processor for executing the computer program to implement the method of any one of claims 1 to 7.
10. A readable storage medium, characterized by, The computer program product comprises:
11. A computer program product comprising computer programs / instructions, characterized in that, a memory for storing a computer program; a processor for executing the computer program to implement the method of any one of claims 1 to 7. The computer program product comprises: a memory for storing a computer program; a processor for executing the computer program to implement the method of any one of claims 1 to 7.