Image generation method, image generation model training method, and related apparatus

Abstract

An image generation method, an image generation model training method, and a related apparatus. The image generation method comprises: acquiring input information; on the basis of the input information, determining inference prompt information; and inputting the input information and the inference prompt information into an image generation model to obtain a token sequence of a target image, the token sequence of the target image comprising a plurality of tokens that correspond to the target image, and the plurality of tokens being generated by the image generation model on the basis of the input information and the inference prompt information. In the present application, when predicting the next token, the image generation model considers distribution information of the target image, and senses the distribution information of the target image, improving the stability of a target image generation process and the accuracy of an image generation result, and ensuring the quality of the generated target image. When processing complex image generation tasks, the present application improves the accuracy and stability of target images in terms of structure and details.

Description

Image generation methods, training methods for image generation models, and related devices

[0001] This application claims priority to Chinese Patent Application No. 2025102032132, filed on February 21, 2025, entitled "Image Generation Method, Training Method for Image Generation Model and Related Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence technology, and in particular to image generation methods, training methods for image generation models, and related apparatus. Background Technology

[0003] In the field of image generation, autoregressive models generate images by predicting tokens one by one. In autoregressive image generation techniques, the model predicts the next token based on the generated tokens and conditional information. In other words, the prediction process for the next token is related to the generated tokens and conditional information, which limits the accuracy and stability of the generated image. Therefore, a technical solution is needed to improve the accuracy and stability of the generated image. Summary of the Invention

[0004] To address the aforementioned issues, this application provides an image generation method, an image generation model training method, and related apparatus to improve the accuracy and stability of image generation results.

[0005] Therefore, the following technical solution is adopted in this application:

[0006] In a first aspect, this application provides an image generation method, comprising: acquiring input information; determining inference prompts based on the input information; and inputting the input information and the inference prompts into an image generation model to obtain a label sequence of a target image, wherein the label sequence of the target image includes multiple labels corresponding to the target image, and the multiple labels are generated by the image generation model based on the input information and the inference prompts. The inference prompts are used to indicate the distribution information of the image during the image generation process.

[0007] In this application, by constructing inference prompts for the target image, which indicate the distribution information of the target image, the image generation model predicts the next label based on the input information and the distribution information of the target image indicated by the inference prompts during the prediction of labels for the target image. Thus, the image generation model considers the distribution information of the target image when predicting the next label, ensuring that it perceives the distribution information of the target image when generating each label, thereby enhancing its contextual reasoning ability during the generation of the next label. This technical solution, which adds an inference step to the label prediction process, improves the stability of the target image generation process, increases the accuracy of the generated image results, and ensures the quality of the generated target image. When handling complex image generation tasks, it improves the accuracy of the target image in terms of structure and detail reconstruction, further enhancing the accuracy of the complex image generation results. This application does not require modification of the image generation model's structure, improving the convenience for users to complete image generation tasks and enhancing the user experience.

[0008] In one possible implementation, the inference prompt information includes category prompt information. The aforementioned determination of the inference prompt information based on the input information includes: determining category prompt information based on the target category indicated by the input information; wherein the target category represents the category of the target image, and the category prompt information represents the distribution information of images within the target category.

[0009] In this implementation, the category hint information represents the distribution information of images belonging to the target category corresponding to the target image. During the image generation model's prediction of each label, the category hint information is embedded to indicate that the target image should possess the distribution information of images belonging to the target category, thus improving the accuracy and stability of the generated image labels and target images.

[0010] In one possible implementation, determining category prompt information based on the target category indicated by the input information includes: determining a reference image belonging to the target category based on the target category indicated by the input information; generating distribution information of the reference image based on the reference image; and obtaining category prompt information based on the distribution information of the reference image.

[0011] In this implementation, one or more reference images of the same category are determined based on the target category corresponding to the target image indicated by the condition information. These reference images and the target image belong to the same category and have the same image distribution characteristics. Therefore, the distribution information of the reference images can be used as a hint for the distribution information of the target image, thereby ensuring that the distribution information of the generated target image conforms to the distribution characteristics of the corresponding category of images and improving the accuracy of the target image.

[0012] In one possible implementation, generating the distribution information of the reference image based on the reference image includes: performing image word segmentation on the reference image to obtain a tag sequence of the reference image; and obtaining the distribution information of the reference image based on the tag sequence of the reference image.

[0013] In this implementation, each marker in the reference image represents information about an image block or a pixel in the reference image, and the marker sequence of the reference image represents information about all image blocks or all pixels in the reference image, that is, the marker sequence of the reference image represents the distribution information in the reference image.

[0014] In one possible implementation, the inference prompt information includes general prompt information. The aforementioned determination of the inference prompt information based on the input information includes: determining general prompt information based on the distribution information of images in multiple categories, where the general prompt information represents the distribution information of images in multiple categories, including the target category.

[0015] In this implementation, multiple categories include the target category corresponding to the target image, and general cue information represents the distribution information of images across multiple categories. During the image generation model's prediction of each label, the general cue information is embedded to indicate that the target image generation process references the distribution information of images across multiple categories. This avoids overfitting of the image generation model to its input information, improves the generalization of the target images generated by the model, and ensures the richness and diversity of the target images.

[0016] In one possible implementation, determining the general prompt information based on the distribution information of images in multiple categories includes: selecting at least one tag from a tag library as the general prompt information, wherein the tag library includes tag sequences corresponding to images in multiple categories.

[0017] In this implementation, the tag library includes tag sequences corresponding to images in multiple categories. At least one tag can be selected from the tag library as general cue information. The general cue information can characterize the distribution information of images in multiple categories, prompting the image generation model to refer to the distribution information generally possessed by images of multiple categories indicated by the general cue information when predicting the next tag.

[0018] In one possible implementation, the above-mentioned selection of at least one tag from the tag library as a general prompt message includes: using a uniformly distributed sampling method to select at least one tag from the tag library as a general prompt message, wherein the uniformly distributed sampling method means that the probability of selecting each tag from the tag library is equal.

[0019] In this implementation, the uniformly distributed sampling method means that the probability of selecting each tag from the tag library is equal. In this way, by using the uniformly distributed sampling method, at least one tag is selected from the tag library as general prompt information, which ensures the balance of the distribution information indicated by the general prompt information across multiple categories, ensures the generalization effect of the general prompt information on the generated target image, and ensures the richness and diversity of the target image.

[0020] In one possible implementation, the method further includes: performing image word segmentation on the input information to obtain a labeled sequence of the input information; and performing image word segmentation on the inference prompt information to obtain a labeled sequence of the inference prompt information. The above-mentioned inputting the input information and inference prompt information into the image generation model includes: inputting the labeled sequence of the input information and the labeled sequence of the inference prompt information into the image generation model.

[0021] In this implementation, the input information to the image generation model can be a sequence of labeled input information and a sequence of labeled inference prompts. Furthermore, the image generation model can predict the next label based on the labeled sequence of conditional information, the labeled sequence of inference prompts, and the already generated label sequence. The label sequence is a data type that the image generation model can directly process, making it easier for the model to handle and resulting in a smaller number of parameters.

[0022] In one possible implementation, the input information includes category information, which represents the category of the target image; and / or, the input information includes text information, which represents the descriptive information of the target image.

[0023] In this implementation, the input information may include at least one of category information and text information, making the method applicable to various fields such as category-based image generation and text-based image generation, thus expanding the application scope of the image generation method. Category information and text information are merely examples of input information; the input information can be any user-inputted data such as instructions, conditions, or text for image generation, and this application does not impose any limitations on this.

[0024] In one possible implementation, the target image's label sequence includes n labels, where the i-th label is generated by the image generation model based on input information, inference prompts, and the previously generated (i-1) labels, where n is a positive integer greater than 1 and i is a positive integer greater than 1 and less than or equal to n.

[0025] In this implementation, the image generation model can be understood as an autoregressive model. The autoregressive model can predict the next label based on the already generated labels. Furthermore, in generating the label sequence of the target image one by one, the autoregressive model also considers the distribution information indicated by the inference prompts, further improving the accuracy of the target images generated by the autoregressive model and ensuring the quality of the generated target images.

[0026] Secondly, this application provides a training method for an image generation model, comprising: constructing a training dataset, the training dataset including target image samples, ground truth values ​​of the labeled sequences of the target image samples, conditional samples of the target image samples, and inference hint samples of the target image samples, wherein the ground truth values ​​of the labeled sequences are obtained by image word segmentation of the target image samples, the conditional samples represent the relevant conditions of the generated image, and the inference hint samples are used to indicate the distribution information of the generated image; inputting the conditional samples and the inference hint samples into the image generation model to obtain the predicted values ​​of the labeled sequences of the generated image, wherein the predicted values ​​of the labeled sequences include multiple labels, which are generated by the image generation model based on the conditional samples and the inference hint samples; determining a loss value based on the ground truth values ​​and the predicted values ​​of the labeled sequences; and adjusting the parameters of the image generation model based on the loss value.

[0027] In one possible implementation, the inference hint samples include category hint samples. The process of constructing category hint samples includes: determining category hint samples based on the category of the target image sample, wherein the category hint samples represent the distribution information of images within the category of the target image sample.

[0028] In one possible implementation, determining the category hint sample based on the category of the target image sample includes: determining a reference image sample based on a conditional sample, wherein the reference image sample is a different image of the same category as the target image sample; performing image word segmentation on the reference image sample to obtain a label sequence of the reference image sample; and using the label sequence of the reference image sample as the category hint sample.

[0029] In one possible implementation, the inference cue samples include general cue samples. The process of constructing general cue samples includes: determining general cue samples based on the distribution information of images in multiple categories, wherein the general cue samples represent the distribution information of images in multiple categories, and the multiple categories include the category of the target image sample.

[0030] In one possible implementation, determining a general cue sample based on the distribution information of images in multiple categories includes: constructing or acquiring an image dataset of multiple categories; generating or acquiring a label sequence of images in the image dataset; and selecting at least one label from the label sequence of images in the image dataset as a general cue sample.

[0031] Thirdly, this application provides an image generation apparatus, comprising: an acquisition module for acquiring input information; a processing module for determining inference prompts based on the input information, the inference prompts being used to indicate the distribution information of the image during the image generation process; and inputting the input information and the inference prompts into an image generation model to obtain a label sequence of a target image, the label sequence of the target image including multiple labels corresponding to the target image, and the multiple labels being generated by the image generation model based on the input information and the inference prompts.

[0032] In one possible implementation, the inference prompt information includes category prompt information. Specifically, the processing module described above is used to: determine category prompt information based on the target category indicated by the input information; wherein the target category represents the category of the target image, and the category prompt information represents the distribution information of images within the target category.

[0033] In one possible implementation, the aforementioned processing module is specifically used to: determine a reference image belonging to the target category based on the target category indicated by the input information; generate distribution information of the reference image based on the reference image; and obtain category prompt information based on the distribution information of the reference image.

[0034] In one possible implementation, the aforementioned processing module is specifically used to: perform image word segmentation on the reference image to obtain a tag sequence of the reference image; and obtain the distribution information of the reference image based on the tag sequence of the reference image.

[0035] In one possible implementation, the inference prompt information includes general prompt information. Specifically, the aforementioned processing module is used to: determine general prompt information based on the distribution information of images in multiple categories, whereby the general prompt information represents the distribution information of images in multiple categories, including the target category.

[0036] In one possible implementation, the aforementioned processing module is specifically used to: select at least one tag from a tag library as general prompt information, wherein the tag library includes tag sequences corresponding to images in multiple categories.

[0037] In one possible implementation, the aforementioned processing module is specifically used to: select at least one tag from the tag library using a uniformly distributed sampling method as general prompt information, wherein the uniformly distributed sampling method means that the probability of selecting each tag from the tag library is equal.

[0038] In one possible implementation, the processing module is further configured to: perform image word segmentation on the input information to obtain a labeled sequence of the input information; and perform image word segmentation on the inference prompt information to obtain a labeled sequence of the inference prompt information. Specifically, the aforementioned processing module is configured to: input the labeled sequence of the input information and the labeled sequence of the inference prompt information into the image generation model.

[0039] In one possible implementation, the input information includes category information, which represents the category of the target image; and / or, the input information includes text information, which represents the descriptive information of the target image.

[0040] In one possible implementation, the target image's label sequence includes n labels, where the i-th label is generated by the image generation model based on input information, inference prompts, and the previously generated (i-1) labels, where n is a positive integer greater than 1 and i is a positive integer greater than 1 and less than or equal to n.

[0041] Fourthly, this application provides a training apparatus for an image generation model, comprising: a training dataset construction module for constructing a training dataset, the training dataset including target image samples, ground truth values ​​of the label sequences of the target image samples, conditional samples of the target image samples, and inference hint samples of the target image samples, wherein the ground truth values ​​of the label sequences are obtained by image word segmentation of the target image samples, the conditional samples represent the relevant conditions of the generated image, and the inference hint samples are used to indicate the distribution information of the generated image; a model training module for inputting the conditional samples and the inference hint samples into the image generation model to obtain the predicted values ​​of the label sequences of the generated image, wherein the predicted values ​​of the label sequences include multiple labels, which are generated by the image generation model based on the conditional samples and the inference hint samples; and determining a loss value based on the ground truth values ​​and the predicted values ​​of the label sequences; and adjusting the parameters of the image generation model based on the loss value.

[0042] In one possible implementation, the inference hint samples include category hint samples. The training dataset construction module described above is specifically used to: determine category hint samples based on the category of the target image sample; the category hint samples represent the distribution information of images within the category of the target image sample.

[0043] In one possible implementation, the training dataset construction module described above is specifically used to: determine reference image samples based on conditional samples, wherein the reference image samples are different images of the same category as the target image samples; perform image word segmentation on the reference image samples to obtain a label sequence of the reference image samples; and use the label sequence of the reference image samples as category cue samples.

[0044] In one possible implementation, the inference hint samples include general hint samples. The aforementioned training dataset construction module is specifically used to: determine general hint samples based on the distribution information of images in multiple categories, whereby the general hint samples represent the distribution information of images in multiple categories, including the category of the target image sample.

[0045] In one possible implementation, the training dataset construction module described above is specifically used to: construct or acquire image datasets of multiple categories; generate or acquire label sequences of images in the image dataset; and select at least one label from the label sequences of images in the image dataset as a general cue sample.

[0046] Fifthly, this application provides a chip system including a processor and a power supply circuit, the power supply circuit being used to supply power to the processor, and the processor being used to execute the method described in any possible implementation of the first aspect and / or the method described in any possible implementation of the second aspect.

[0047] Sixthly, this application provides a computing device including a processor and a memory. The processor is configured to execute instructions stored in the memory to cause the computing device to perform the methods described in any possible implementation of the first aspect and / or the methods described in any possible implementation of the second aspect.

[0048] In a seventh aspect, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster performs the method described in any possible implementation of the first aspect and / or the method described in any possible implementation of the second aspect.

[0049] Eighthly, this application provides a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in any possible implementation of the first aspect and / or the method described in any possible implementation of the second aspect, wherein the cluster of computing devices includes at least one computing device.

[0050] Ninthly, this application provides a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method described in any possible implementation of the first aspect and / or the method described in any possible implementation of the second aspect, wherein the cluster of computing devices includes at least one computing device.

[0051] It is understood that the beneficial effects of aspects two through nine above can be found in the relevant descriptions in aspect one above, and will not be repeated here. Attached Figure Description

[0052] The accompanying drawings used in the embodiments or technical description are briefly introduced below.

[0053] Figure 1 is a schematic diagram of the composition of an image generation system provided in an embodiment of this application;

[0054] Figure 2 is a flowchart illustrating an image generation method provided in an embodiment of this application;

[0055] Figure 3 is a flowchart illustrating a training method for an image generation model provided in an embodiment of this application;

[0056] Figure 4 is a block diagram illustrating the training process of an image generation model provided in an embodiment of this application;

[0057] Figure 5 is a schematic diagram of an example of the image generation method provided in the embodiments of this application;

[0058] Figure 6 is a schematic diagram of the image generation model predicting the next labeling process provided in the embodiment of this application;

[0059] Figure 7 is a schematic diagram of the results of an ablation study of the reasoning prompt information provided in the embodiments of this application;

[0060] Figure 8 is a schematic diagram of the composition of an image generation device provided in an embodiment of this application;

[0061] Figure 9 is a schematic diagram of the composition of a training device for an image generation model provided in an embodiment of this application;

[0062] Figure 10 is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0063] Figure 11 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0064] Figure 12 is a schematic diagram of another computing device cluster provided in an embodiment of this application. Detailed Implementation

[0065] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0066] In this article, the term "and / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone. The symbol " / " in this article indicates that the related objects have an "or" relationship; for example, A / B means A or B.

[0067] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0068] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0069] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc. "Several" means one or more.

[0070] To facilitate understanding of the solutions provided in the embodiments of this application, some of the terms involved in this solution will be briefly introduced first.

[0071] Large Language Models (LLMs): Large language models are natural language processing (NLP) models trained using deep learning techniques. They are capable of understanding, generating, and manipulating human language. These models learn language patterns, grammatical rules, semantic understanding, and logical reasoning abilities by processing and learning from large amounts of text data. They are widely used in tasks such as text generation, translation, dialogue systems, and question answering. Large language models are also known as large models or large language models.

[0072] An autoregressive model (AR) is a statistical model that uses past data to predict future values. Neural network models can be used as a tool or method to implement autoregressive models; for example, Large Language Modeling (LLM) can be used as an implementation of an autoregressive model. An autoregressive model assumes that the current value can be represented by values ​​from a series of previous time points according to a certain relationship. In an autoregressive model, "auto" means that the model relies only on its own historical data and does not require other external information; "regressive" means that the model finds the correlation between variables by establishing mathematical relationships. For example, an autoregressive model can predict tomorrow's weather based on the weather over the past few days.

[0073] Image tokenizers, also known as visual tokenizers, are typically used to convert images into a series of discrete tokens for further processing or generation. This architecture is particularly important in image generation tasks because it allows models to progressively generate images in an autoregressive manner. Image tokenizers are usually trained on deep learning models that encode images into a series of discrete tokens. For example, an image tokenizer can be implemented using a Convolutional Neural Network (CNN) architecture to segment the image into small blocks and encode each block as a discrete token.

[0074] Token library: Also known as a codebook. A token library contains a large visual vocabulary, with each word corresponding to a local feature or pattern in an image. In practical applications, existing token libraries are typically used in conjunction with pre-trained neural network models to achieve image tokenization. For example, a CNN network model segments an image into small patches and selects tokens from a predefined codebook to encode each image patch.

[0075] Tokens: In natural language processing, tokens are typically defined as words or characters. In image processing, tokens can be image patches, pixel values, or other forms of discrete representation.

[0076] The "next tag prediction" approach of AR models is widely used in image processing and natural language processing. Its basic idea is to progressively predict the next tag based on an existing tag sequence. AR models learn the dependencies between tags through training data. During training, the AR model attempts to predict the next tag for a given tag sequence. The training data typically consists of real tag sequences, and the AR model adjusts its parameters based on these sequences to minimize prediction errors. During generation, the AR model iteratively predicts the next tag based on the existing tag sequence until a complete sequence is generated. The AR model considers previously generated tags when generating each tag, enabling it to generate context-dependent sequences. In practical applications, image segmentation architectures and the "next tag prediction" approach of AR models are often used in combination. For example, in the training phase of the autoregressive model, the visual tagger first converts the image into a tag sequence; then, the autoregressive model predicts the next tag one by one according to the "next tag prediction" method; the loss value is calculated based on the tag sequence predicted by the autoregressive model and the tag sequence converted by the visual tagger; and the parameters of the autoregressive model are adjusted based on the loss value. In the inference phase of the autoregressive model, the autoregressive model predicts the next tag one by one according to the "next tag prediction" method, combined with the input conditional information, and gradually generates the image based on these tag sequences.

[0077] Conditional information refers to the information added to the AR model that guides the image generation process, such as using conditional tokens to guide image generation. Incorporating conditional information into the AR model can be achieved through strategies such as: conditional embedding, embedding conditional information (such as category labels, text descriptions, etc.) into a low-dimensional space, for example, through simple fully connected layers or embedding layers; fusion embedding, fusing conditional embeddings with pixel features, for example, through simple concatenation or weighted summation, such as concatenating the conditional embeddings with the features of the current pixel and its surrounding pixels before generating each pixel; and modifying the AR model structure to ensure that the AR model can accept and process the fused features, which may involve modifying the number of input channels in convolutional layers or considering conditional information when generating each pixel.

[0078] Autoregressive raster generation order refers to the sequence in which an autoregressive model generates pixels or markers in an image from left to right and top to bottom during image generation. While a common strategy for autoregressive models in image generation, it is not the only option. For example, random ordering or other more complex generation strategies can be used to improve the generation order of the autoregressive model.

[0079] Chain of Thought (COT): This term describes a reasoning mechanism used by Large Language Models (LLMs) when processing tasks. In Natural Language Processing (NLP), a model might understand complex sentences or solve mathematical problems through a step-by-step construction process, similar to a human thought chain. By simulating this step-by-step reasoning, Artificial Intelligence (AI) systems can more effectively understand and respond to complex problems.

[0080] In the field of image generation, autoregressive models generate images by predicting tokens one by one. In autoregressive image generation techniques, the model predicts the next token based on the generated tokens and conditional information. In other words, the prediction process for the next token is related to the generated tokens and conditional information, which limits the accuracy and stability of the generated image. Therefore, a technical solution is needed to improve the accuracy and stability of the generated image.

[0081] In view of this, embodiments of this application provide an image generation method. In this image generation method, inference prompts for a target image are constructed, the inference prompts being used to indicate the distribution information of the target image; conditional information (also referred to as input information) and the inference prompts are input into an image generation model; guided by the conditional information and the inference prompts, the image generation model predicts the labels of the target image one by one, and generates the target image based on the predicted label sequence of the target image.

[0082] In this embodiment, by constructing inference prompts for the target image, which indicate the distribution information of the target image, the image generation model predicts the next label based on the conditional information and the distribution information of the target image indicated by the inference prompts during the prediction of labels for each target image. Thus, based on the "next label prediction" paradigm, the image generation model considers the distribution information of the target image when predicting the next label, ensuring that it perceives the distribution information of the target image when generating each label, thereby enhancing the contextual reasoning ability of the next label generation process. This technical solution, which adds a reasoning step to the label prediction process, improves the stability of the target image generation process, increases the accuracy of the image generation results, and ensures the quality of the generated target image.

[0083] In this embodiment, when processing complex image generation tasks, the accuracy of the target image in terms of structure and detail reconstruction is improved, further enhancing the accuracy of the complex image generation results. The image generation model references the distribution information of the target image when generating each marker, thereby enhancing the accuracy and stability of the next marker generation result and the overall target image generation result. Specifically, the target image generation process considers the distribution information of the target image, ensuring the accuracy of the image structure; the accuracy and stability of the marker generation results in the image ensure the accuracy of the image details. Therefore, this embodiment can guarantee the accuracy of the generated target image in terms of structure and detail when processing complex image generation tasks, further improving the accuracy of the complex image generation results.

[0084] In this embodiment, within the "next label prediction" paradigm of the image generation model, inference hints are added during the prediction of the next label, enhancing the image generation model's contextual reasoning ability and the stability of the generated results in complex tasks. Simultaneously, no modification to the structure of the image generation model is required. For example, the image generation method of this embodiment improves image generation performance by approximately 20% compared to related autoregressive image generation methods, significantly enhancing the quality and consistency of the generated image results.

[0085] To better understand the image generation method of this application embodiment, the application scenarios of the image generation method of this application embodiment are described below, but the description of the application scenarios does not constitute a limitation on the embodiments of this application.

[0086] Figure 1 is a schematic diagram of the composition of an image generation system provided in an embodiment of this application. As shown in Figure 1, in this embodiment, an image generation system 100 includes a terminal 110 and a server 120. For example, the terminal 110 obtains the user's condition information and sends it to the server 120; the server 120 deploys an image generation model, which generates a target image based on the image generation method of this embodiment. Optionally, the server 120 sends the target image to the user; the user obtains the target image through the terminal 110. The image generation method of this embodiment adds inference prompt information during the prediction of the next marker by the image generation model, which considers the distribution information of the target image in the prediction of the next marker, improving the accuracy and stability of the image generation result.

[0087] Optionally, server 120 may include one or more servers (Figure 1 illustrates this by including one server as an example), and server 120 may provide the methods or apparatus provided in the embodiments of this application to one or more terminals.

[0088] Optionally, a relevant application may be installed on the terminal 110. The application or webpage can provide an interface that displays the target image. The terminal 110 can receive conditional information input by the user on the interface and send the conditional information to the server 120. The server 120 is equipped with the image generation model of this application embodiment. In the process of the image generation model predicting the next marker, inference prompts are added, which makes the accuracy and stability of the target image higher.

[0089] It should be understood that in some optional implementations, terminal 110 may deploy the large language model and complete the work on its own without the cooperation of server 120, and this application embodiment is not limited in this regard. In some optional implementations, server 120 may also implement the work based on condition information received from other devices outside the terminal or locally stored, without the cooperation of terminal 110, and this application embodiment is not limited in this regard.

[0090] The product form of terminal 110 in Figure 1 will be described next. In this embodiment, terminal 110 can be a mobile phone, tablet computer, wearable device, in-vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc. This embodiment does not impose any limitations on this.

[0091] It can be further understood that server 120 can be various types of servers, such as x89 architecture servers, specifically rack servers, blade servers, high-density servers, rack servers, or high-performance servers. In other words, this application embodiment does not specifically limit the specific type of server. Furthermore, it can be understood that the server structure shown in Figure 1 does not constitute a limitation on the server structure; a server may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0092] Furthermore, server 120 can be configured as an independent physical server, or as a server cluster or distributed system consisting of multiple physical servers. It can also be configured as a cloud server or cloud server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The cloud server cluster is deployed in several cloud data centers. The software can be an application that implements object control methods, but it is not limited to the above forms.

[0093] In one possible scenario, server 120 can function as a cloud platform (a software platform employing application virtualization technology, integrating multiple functions such as software search, download, use, management, and backup). In practical use, server 120 can deploy a cloud management platform and a data center, with terminal 110 interacting with the cloud through the cloud management platform. Additionally, the data center can deploy nodes, which can be virtual machine instances, container instances, physical servers, etc.

[0094] In another possible scenario, the method provided in this application embodiment can be implemented by software. The software has a terminal and a server; the terminal 110 runs the software, and the server 120 runs the software's server. While the terminal 110 is running the software, it can call the server running on the server 120 to implement the method provided in this application embodiment.

[0095] In other words, the method provided in this application embodiment can be applied to the terminal 110 or the server 120. In specific implementation, it can run as software on the terminal 110 or the server 120; for example, the software can be a service or an application. This application embodiment can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application embodiment can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0096] Furthermore, the system that can implement the technical solution of this application embodiment includes cloud servers, web pages, application program interfaces (APIs), mobile application software, etc. The large language model of this application embodiment can be deployed on the server side, website, or mobile application software to meet the text generation needs of enterprise users and individual users, and can also be provided to developers for secondary development and use through open API interfaces. Furthermore, the implementation of this application embodiment mainly involves deploying the relevant platforms and applications of the large language model of this application embodiment, such as cloud platforms, websites, applications (APPs), etc.

[0097] Figure 2 is a flowchart illustrating an image generation method provided in an embodiment of this application. As shown in Figure 2, this embodiment of the application provides an image generation method, which mainly includes the following steps:

[0098] Step S210: Obtain condition information. This condition information is also referred to as input information.

[0099] In this embodiment, condition information refers to the information of the generated image input in the image generation task; therefore, condition information is also called input information. Condition information represents the conditions for generating the target image. It should be noted that the condition information can be any relevant conditions for generating the target image.

[0100] For example, in the field of text-to-image generation, the conditional information can be any text information, representing the descriptive information of the target image. For instance, the text information could be "generate a picture of a cat eating cat food," and the image generation model could generate a picture of a cat eating cat food based on this text information.

[0101] For example, in the field of category-based image generation, the conditional information can be any category information, which represents the category of the target image. For instance, the category information could be "cat," and the image generation model could generate an image containing a cat based on the "cat" category information.

[0102] Step S220: Determine inference prompts based on the condition information. The inference prompts are used to indicate the distribution information of the target image.

[0103] In this embodiment, during the process of the image generation model predicting each marker, inference prompts are embedded to indicate the distribution information of the target image. This makes the image generation process closer to the contextual scanning and logical reasoning in human visual perception when generating images, thereby improving the accuracy and stability of the image generation results.

[0104] For example, the conditional information for the image generation task is "generate a picture of a cat." The distribution information indicated by the inference prompts could be, for instance, "the upper left part of the image contains the cat's left side of the face, left eye, and left ear; the upper right part contains the cat's right side of the face, right eye, and right ear; the lower left part contains the cat's left side of the body, left upper limb, and left lower limb; the lower right part contains the cat's right side of the body, right upper limb, and right lower limb; and the cat's tail is located in either the lower left or lower right part of the image." Considering this distribution information, the image generation model achieves higher accuracy and stability in generating each marker of the target image, resulting in higher accuracy and stability of the target image itself.

[0105] Furthermore, in the above example of "generating a picture of a cat," the image generation process in this embodiment more closely resembles the contextual scanning and reasoning logic in human visual perception when generating images. For instance, when drawing a cat image, humans consider the distribution information of the target image, such as the distribution information of the cat target image mentioned above, when generating details and local content. With this distribution information considered, the accuracy and stability of each part of the generated target image are higher, reducing the possibility of image generation errors and improving the accuracy and stability of the target image. In other words, this embodiment adds a reasoning step with inference prompts during the image generation model's tag generation process, making image generation closer to the contextual scanning and logical reasoning of human image generation, thus improving the accuracy and stability of the image generation results.

[0106] Optionally, the inference prompts may include at least one of category prompts and general prompts. Category prompts represent the distribution information of images in this category (the target category corresponding to the target image), thereby improving the accuracy and stability of the generated target image in terms of category. General prompts represent the distribution information of images in multiple categories (including the target category corresponding to the target image), thereby improving the generalization of the generated target image and ensuring its richness and diversity.

[0107] It should be noted that the category information in the application scenario of category-based image generation is different from the category hint information in inference hint information. Category information is a word or token that represents the category of the generated target image; category hint information is information that characterizes the distribution of information in the image, indicating the distribution information of the generated target image.

[0108] Optionally, the category hint information can be further explained. The category hint information can be determined based on the target category indicated by the condition information. Here, the target category represents the category of the target image, and the category hint information represents the distribution information of images belonging to the target category to which the target image belongs.

[0109] Furthermore, category hints represent the distribution information of images belonging to a certain class (i.e., the target class corresponding to the target image). By embedding category hints during the image generation model's prediction of each label, the distribution information of the target class that the target image should possess is indicated, thus improving the accuracy and stability of the generated image labels and target images.

[0110] Optionally, the implementation method of category prompt information is further explained. Reference images belonging to the target category can be determined based on the target category indicated by the condition information; distribution information of the reference images can be generated; and category prompt information can be obtained based on the distribution information of the reference images.

[0111] Furthermore, based on the target category corresponding to the target image indicated by the conditional information, one or more reference images of the same category are determined. These reference images and the target image belong to the same category and have the same image distribution characteristics. Therefore, the distribution information of the reference images can be used as a clue to the distribution information of the target image, thereby ensuring that the distribution information of the generated target image conforms to the distribution characteristics of the corresponding category of images and improving the accuracy of the target image.

[0112] Optionally, the process of generating the distribution information of the reference image can be further explained. Image segmentation can be performed on the reference image to obtain a tag sequence; based on the tag sequence, the distribution information of the reference image can be obtained. In other implementations, the tag sequence of the reference image can also be obtained by searching a tag library based on conditional information. A tag library refers to a collection of tag sequences for images belonging to several categories; by searching this tag library based on conditional information, the tag sequence of the reference image matching the conditional information can be obtained.

[0113] Furthermore, each marker in the reference image represents information about an image block or a pixel in the reference image, and the sequence of markers in the reference image represents information about all image blocks or all pixels in the reference image, that is, the sequence of markers in the reference image represents the distribution information in the reference image.

[0114] It should be noted that image segmentation can be implemented using any segmentation technique, and this application does not impose any limitations on this. For example, an image segmenter or any neural network model can be used. An image segmenter can divide an image into multiple image blocks or pixels and encode each image block or pixel into a discrete token.

[0115] Optionally, the implementation method of the general prompt information is further described. Based on the distribution information of images in multiple categories, the general prompt information is determined. The general prompt information represents the distribution information of images in multiple categories, including the target category.

[0116] Furthermore, multiple categories include the target category corresponding to the target image, and general cue information represents the distribution information of images across multiple categories. During the image generation model's prediction of each label, general cue information is embedded to indicate that the target image generation process references the distribution information of images across multiple categories. This avoids overfitting of the image generation model to its input, improves the generalization of the target images generated by the model, and ensures the richness and diversity of the target images.

[0117] Optionally, the process for determining the general prompt information is further explained. A tag library is constructed or obtained, which includes tag sequences corresponding to images in multiple categories; from the tag library, at least one tag is selected as the general prompt information. The tag library includes tag sequences corresponding to images in the aforementioned multiple categories.

[0118] Furthermore, the label library includes label sequences corresponding to images in multiple categories. At least one label can be selected from the label library as general cue information. In this way, the general cue information can characterize the distribution information of images in multiple categories, prompting the image generation model to refer to the distribution information generally possessed by images in multiple categories indicated by the general cue information when predicting the next label.

[0119] For example, the construction process of the token library is illustrated. Each of the multiple categories corresponds to an image dataset. The images in each category's image dataset are processed through image word segmentation to obtain a token sequence for each image in each category's image dataset. In this way, the token sequences of images in the image datasets of multiple categories form a token library, also known as a codebook.

[0120] Optionally, an exemplary description of a method for selecting tags from a tag library is provided. Using a uniformly distributed sampling method, at least one tag is selected from the tag library as general prompt information. The uniformly distributed sampling method implies that each tag has an equal probability of being selected from the tag library.

[0121] Furthermore, the uniformly distributed sampling method means that the probability of selecting each label from the label library is equal. In this way, by using the uniformly distributed sampling method, at least one label is selected from the label library as general cue information, which ensures the balance of the distribution information indicated by the general cue information across multiple categories, ensures the generalization effect of the general cue information on the generated target image, and ensures the richness and diversity of the target image.

[0122] Step S230: Input the conditional information and inference prompts into the image generation model to obtain the label sequence of the target image. The label sequence of the target image includes multiple labels corresponding to the target image, and the multiple labels are generated by the image generation model based on the input information and inference prompts.

[0123] Step S240: Generate the target image based on the label sequence of the target image.

[0124] In this embodiment, when the image generation model predicts the next label, the embedding of inference hints reduces the impact of errors in the already generated labels on the prediction of the next label. The inference hints have a certain corrective effect, improving the accuracy and stability of the image generation results. Compared with methods that predict the next label based on conditional information and already generated labels, the embedding of inference hints results in higher accuracy, less erroneousness, and greater stability in the image generation results.

[0125] Optionally, the target image's label sequence includes n labels, where the i-th label among the n labels is generated by the image generation model based on the input information, inference prompts, and the previously generated i-1 labels, where n is a positive integer greater than 1 and i is a positive integer greater than 1 and less than or equal to n.

[0126] In this optional implementation, the image generation model is an autoregressive model. The autoregressive model can predict the next label based on the generated labels. In generating the label sequence of the target image one by one, the autoregressive model also considers the distribution information indicated by the inference prompts, further improving the accuracy of the target images generated by the autoregressive model and ensuring the quality of the generated target images.

[0127] It is understandable that, in the process of "predicting the next label," an image generation model can refer to already generated labels; that is, the image generation model predicts the next label based on the input conditional information, inference hints, and already generated labels. Alternatively, the image generation model can also predict the next label without referring to already generated labels; that is, the image generation model predicts the next label based solely on the input conditional information and inference hints.

[0128] In other words, whether an image generation model references previously generated labels during the label prediction process can be chosen based on the specific design of the image generation model and the application scenario. For example, in some application scenarios, global or contextual information represented by inference hints may be used to predict labels, rather than relying on previously generated labels. Therefore, setting inference hints during label prediction can more effectively utilize global and contextual information, thereby reducing reliance on previously generated labels.

[0129] Understandably, in this optional implementation, from an algorithmic perspective, the image generation model can be understood as an autoregressive model, generating images through an autoregressive algorithm. From an implementation perspective, the image generation model can be implemented using any program code, functional modules, or other tools or methods. Neural network models can be used as an implementation method for the autoregressive algorithm of the image generation model; for example, the Large Language Model (LLM) can be used as a specific implementation of an autoregressive model.

[0130] It's important to note that the image generation model can be a large language model. The inference hints, category hints, and general hints mentioned above are not equivalent to the hint words generated by the hint engineering of a large language model. Inference hints, category hints, and general hints are part of the input data of the large language model, while the hint words generated based on hint engineering aim to help the image generation model better understand the input data. For example, in practical applications, conditional information is input into the image generation model in the form of tokens. Inference hints, category hints, and general hints are also input into the image generation model in the form of tokens, concatenated with the conditional information.

[0131] Taking inference prompts as an example, prompt words can be generated based on inference prompts and conditional information. A prompt word could be something like, "Please predict the labels of the target image one by one based on the following information, and generate the target image based on all predicted labels. The condition for this image is [conditional information]. In the process of predicting the next label based on [conditional information] and the already predicted labels, please consider the distribution information in the [inference prompts] to improve the layout effect of the target image."

[0132] For example, an exemplary description is given of the data types input to the image generation model.

[0133] For example, an image generation model can include functional modules such as an image segmenter, an image segmentation module, and an image segmentation layer. In this case, the input to the image generation model can be the aforementioned conditional information and inference prompts. The image segmentation module within the image generation model performs image segmentation on the conditional information and inference prompts, obtaining a labeled sequence for the conditional information and a labeled sequence for the inference prompts. The image generation model can then predict the next label based on the labeled sequence of the conditional information, the labeled sequence of the inference prompts, and the already generated labeled sequence.

[0134] For example, in addition to the image generation model, there can be image segmentation modules, such as image word segmenters, image word segmentation modules, and image word segmentation models, that have image word segmentation functionality. In this case, the image word segmentation module can be used to perform image word segmentation processing on the conditional information and the inference prompt information, obtaining the labeled sequences of the conditional information and the inference prompt information. The input to the image generation model can be the labeled sequences of the conditional information and the inference prompt information. Furthermore, the image generation model can predict the next label based on the labeled sequences of the conditional information, the labeled sequences of the inference prompt information, and the already generated labeled sequences.

[0135] For example, the process of decoding and generating a target image based on a token sequence is illustrated. The decoding of the token sequence is typically accomplished using a decoder. This decoder accepts the token sequence as input and outputs an image or a representation of the image. This decoder can be a Transformer model, a specially designed image generation network, or a neural network block or layer within the aforementioned image generation model.

[0136] Furthermore, the decoding process of the token sequence can be understood as an image reconstruction process. During the decoding phase, the decoder uses information from the token sequence to reconstruct the image. This may involve mapping the tokens back to the image pixel space, or using the tokens to guide the generation process, such as generative models based on Generative Adversarial Networks (GANs).

[0137] Optionally, the decoded target image may undergo further optimization and refinement steps to improve image quality. Optimization and refinement steps may include adjusting the weights of the tokens, applying post-processing filters, or using other techniques to enhance image details.

[0138] In addition to the image generation model involved in the above-described image generation system 100 and image generation method, this application embodiment also provides a training method for the image generation model.

[0139] Figure 3 is a flowchart illustrating a training method for an image generation model provided in an embodiment of this application. As shown in Figure 3, the training method for an image generation model provided in this embodiment mainly includes the following steps:

[0140] Step S310: Construct the training dataset. The training dataset includes target image samples, ground truth values ​​of the label sequences of the target image samples, conditional samples of the target image samples, and inference hint samples of the target image samples. The label sequences of the target image samples are obtained by image word segmentation of the target image samples. The conditional samples represent the relevant conditions of the generated images, and the inference hint samples are used to indicate the distribution information of the generated images. The generated images refer to the images generated during the training phase based on the labels predicted one by one by the image generation model.

[0141] Optionally, the ground truth values ​​of the label sequence of the target image sample are the labels of the training process, which can be selected from an existing label library or obtained by performing image word segmentation on the target image sample.

[0142] In this embodiment, the conditional sample of the target image sample can be understood as the conditional information of the target image sample. For details regarding the conditional information, please refer to the above description, which will not be repeated here.

[0143] In this embodiment, the inference hint sample of the target image sample can be understood as the inference hint information of the target image sample. For relevant explanations regarding the inference hint information, please refer to the above description, which will not be repeated here.

[0144] Optionally, the inference hint samples include at least one of category hint samples and general hint samples.

[0145] Optionally, the process of constructing the category hint sample includes: determining a category hint sample based on the category of the target image sample, wherein the category hint sample is used to indicate the distribution information of images in the category of the target image sample. For example, a reference image sample is determined based on a conditional sample; the reference image sample is a different image of the same category as the target image sample; image segmentation is performed on the reference image sample to obtain a tag sequence of the reference image sample; and the tag sequence of the reference image sample is used as the category hint sample.

[0146] Furthermore, the category hint sample can be understood as a sample of category hint information. For relevant explanations of category hint information, please refer to the above description, which will not be repeated here.

[0147] Optionally, the process of constructing a general cue sample includes: determining a general cue sample based on the distribution information of images in multiple categories, wherein the general cue sample represents the distribution information of images in multiple categories, and the multiple categories include the category of the target image sample. For example, constructing or obtaining an image dataset of multiple categories; generating a label sequence of images in the image dataset; and selecting at least one label from the label sequence of images in the image dataset as a general cue sample.

[0148] Furthermore, the general prompt sample can be understood as a sample of general prompt information. For relevant explanations of general prompt information, please refer to the above description, which will not be repeated here.

[0149] Step S320: Input the conditional samples and inference cue samples into the image generation model to obtain the predicted value of the label sequence of the generated image. The predicted value of the label sequence includes multiple labels, which are generated by the image generation model based on the conditional samples and inference cue samples.

[0150] Step S330: Determine the loss value based on the true value of the labeled sequence and the predicted value of the labeled sequence.

[0151] For example, the loss value can be calculated by determining a loss function, such as using a cross-entropy loss function. This loss value measures the difference between the predicted labels of the image generation model and the true labels. For each predicted label, the cross-entropy between its predicted probability distribution and the true label is calculated, and then the loss values ​​of the cross-entropy of all predicted labels in the image are summed to obtain the loss value for the entire training sample.

[0152] Step S340: Adjust the parameters of the image generation model based on the loss value.

[0153] It should be noted that image generation models can be trained using any method, such as single-sample training or batch training. For single-sample training, the loss values ​​of all predicted labels for a single target image sample are summed to obtain the total loss value for the entire target image sample. This loss value is used to measure the prediction accuracy of the image generation model on the current target image sample. For batch training, the average loss of all target image samples in a batch is calculated as the batch loss, which is used to measure the prediction accuracy of the image generation model on the current batch of data.

[0154] Figure 4 is a block diagram illustrating the training process of an image generation model provided in an embodiment of this application. As shown in Figure 4, during the training process of the image generation model, the training dataset includes target image samples, ground truth values ​​of the labeled sequences of the target image samples, conditional samples of the target image samples, and inference cue samples of the target image samples.

[0155] Optionally, the ground truth of the labeled sequence is obtained by performing image segmentation on the target image sample. For example, if the target image sample is a picture of a cat, the ground truth of the labeled sequence is the labeled sequence of this cat picture.

[0156] Optionally, the conditional sample of the target image sample refers to the conditions related to the target image sample. For example, the conditional sample can be a label or text generated from the target image sample. For instance, if the target image sample is a picture of a cat, the conditional sample could be "generate a picture of a cat".

[0157] Optionally, the inference cue samples further include at least one of category cue samples and general cue samples. For example, the target image sample is a picture of a cat, and the process of determining category cue samples and general cue samples is further explained.

[0158] Optionally, the process of determining the category hint sample includes: determining a reference image sample based on the condition sample, for example, the reference image sample is a picture of a cat that is different from the target image sample; performing image segmentation on the reference image sample to obtain a tag sequence of the reference image sample; and using the tag sequence of the reference image sample as the category hint sample. In other implementations, the tag sequence of the reference image sample can also be obtained by searching a tag library based on the condition sample. The tag library refers to a collection of tag sequences of images including several categories; by searching this tag library based on the condition sample, the tag sequence of the reference image sample that matches the condition sample can be obtained.

[0159] Optionally, the process of determining the general cue sample includes: determining the category of the target image sample as "cat" based on the condition sample; constructing or obtaining an image dataset with multiple categories, including the category "cat", such as "cat", "dog", "sheep", etc.; the label library of this image dataset includes the label sequence of each image; and selecting some labels from the label sequence of the label library of this image dataset as the general cue sample.

[0160] In one example, a picture of a cat is used as the target image sample for illustration. The conditional sample is the user input "generate a picture of a cat". From the image dataset, a picture of a cat is selected as the target image sample. From the image dataset, a different picture of a cat is selected as a reference image sample; image segmentation is performed on the reference image sample to obtain its tag sequence; this tag sequence is used as a category hint sample. The category of the target image sample is "cat"; multiple image datasets corresponding to multiple categories are constructed or obtained, where multiple categories include "cat", and other categories include "cat", "dog", "sheep", etc.; the tag sequences of the images in this image dataset are constructed or obtained; from the tag sequences of the images in this image dataset, a portion of the tags are selected as a general hint sample.

[0161] In this example, conditional samples, category cue samples, and general cue samples are input into the image generation model. As the image generation model predicts labels one by one, its input consists of the conditional samples, category cue samples, general cue samples, and the already predicted labels. Based on all the labels predicted by the image generation model, the predicted values ​​of the label sequence are determined. The loss value is calculated by combining the ground truth value of the label sequence corresponding to the target image sample with the predicted value of the label sequence predicted by the image generation model. The parameters of the image generation model are then adjusted based on the loss value.

[0162] To further illustrate the technical solutions of the embodiments of this application, an example is provided to further explain the image generation system 100 and the image generation method described above. This example is intended to exemplify the technical solutions of the embodiments of this application and does not constitute a limitation on the technical solutions of the embodiments of this application.

[0163] Figure 5 is a schematic diagram of an example of the image generation method provided in this application. As shown in Figure 5, in one example, the input to the image generation model includes conditional information and inference prompts. The image generation model then predicts the next label of the target image one by one based on the conditional information, inference prompts, and the generated labels, ultimately obtaining a label sequence of the target image. Using a decoder, the target image is generated based on the label sequence of the target image.

[0164] In this example, the conditional information can be text information from the application scenario of text-based image generation, such as "generate a picture of a locust"; the conditional information can also be class information from the application scenario of category-based image generation, such as "locust". The target image output by the image generation model is one or more images containing "locust", for example, taking the first two images in the generated results as the output of the image generation model.

[0165] Optionally, the reasoning hints may include at least one of category hints and general hints.

[0166] Optionally, during the determination of category hints, the category of the target image can be determined based on the conditional information; then, one or more different reference images of the same category as the target image can be found; and the distribution information of the reference images can be used as category hints. For example, image segmentation can be performed on the reference images to obtain a label sequence of the reference images, and the label sequence of the reference images can be used as category hints.

[0167] For example, the conditional information could be textual information from a text-based image generation scenario, such as "generate a picture of a locust." If the target category indicated by the conditional information is "locust," then a reference image containing "locust," different from the target image, can be found. For instance, image segmentation processing can be performed on the reference image containing "locust" to obtain a tag sequence of the reference image containing "locust," and this tag sequence can be used as category prompt information in the target image generation process.

[0168] Optionally, in determining the general prompt information, the general prompt information can be determined based on the distribution information of images in multiple categories that contain the target image's target category. For example, the tag library includes tags corresponding to tag sequences of images in multiple categories. At least one tag is selected from the tag library to form a tag sequence as the general prompt information. For example, a uniformly distributed sampling method can be used to select at least one tag from the tag library.

[0169] For example, based on the conditional information, the target category of the target image is determined to be "locust". Further, the distribution information of multiple categories including "locust" is determined, such as "ant", "locust", "locust", "spider", etc. Images of multiple categories can be from existing image datasets, or they can be constructed or obtained from image datasets of multiple categories based on the target category indicated by the input conditional information. Image segmentation processing technology can be used to convert the images of multiple categories into a tag form, thereby obtaining a tag library. The tag library can be a codebook corresponding to an existing image dataset, or it can be a codebook for a constructed image dataset.

[0170] In this example, the conditional information is the class information in the application scenario of the category-generated image: "locust". Analyzing the conditional information, the target class indicated is "locust". Then, a reference image containing "locust" can be found, different from the target image. For example, image segmentation processing can be performed on the reference image containing "locust" to obtain a tag sequence of the reference image containing "locust". This tag sequence is used as the category hint information in the target image generation process. Further, the distribution information of multiple categories including "locust" is determined, such as "ant", "locust", "cricket", "spider", etc. Through image segmentation processing technology, images of multiple categories are converted into tag form, thus obtaining a tag library. From the tag library, at least one tag is selected to form a tag sequence, which serves as general hint information. The inference hint information can include at least one of category hint information and general hint information. The conditional information "locust" label and the inference prompt information label are concatenated and input into the image generation model. Under the guidance of the conditional information and the inference prompt information, the image generation model predicts the label of the target image one by one, obtains the label sequence of the target image, and generates the target image based on the label sequence of the target image through the decoder, as shown in Figure 5, which contains the target image "locust".

[0171] In this embodiment, while autoregressive image generation technology can generate high-quality images based on the "next label prediction" paradigm of image generation models, it faces difficulties in reconstructing image structure and details when handling complex image generation tasks. Furthermore, its generation process differs from contextual scanning and logical reasoning in human visual perception, limiting the accuracy and stability of the generated results. In human visual perception, the reasoning sequence is global. For example, in generating an image of a "locust," humans already know the approximate layout of the "locust," resulting in more accurate and consistent "locust" images with better stability. This embodiment borrows from the human image generation process, performing exceptionally well in handling complex tasks. It improves the stability and accuracy of the reasoning process by introducing intermediate reasoning cues.

[0172] Figure 6 is a schematic diagram of the image generation model predicting the next label according to an embodiment of this application. As shown in Figure 6, in the first generation step, the input of the image generation model is conditional information and inference hints, and the output is the first label; in the second generation step, the input of the image generation model is conditional information, inference hints, and the generated label (the first label), and the output is the second label; in the third generation step, the input of the image generation model is conditional information, inference hints, and the generated labels (the first label and the second label), and the output is the third label; until all the labels that need to be predicted are completed, the label sequence of the target image is obtained. The label sequence is, for example, a 9-row, 7-column label sequence as shown in Figure 6.

[0173] For example, the generation order of the marker sequence can be the autoregressive raster generation order, that is, from left to right and from top to bottom, or it can be a random order or other more complex order strategy. This application embodiment does not limit this.

[0174] In this embodiment, inference hints are added when the image generation model generates each tag. This makes the tag prediction process similar to a chain-of-thought (COT) generation method. In this embodiment, when the image generation model predicts the next tag, the inputs are conditional information, inference hints, and the already generated tags. Similar to the step-by-step inference process of COT, each step of the inference process considers the already generated inference results.

[0175] The image generation method in this application embodiment can be understood as an autoregressive image generation method based on chain-like thinking. In this application embodiment, the prediction process of the next label is set as the inference process of COT (Contextuality-Oriented Reasoning). Introducing COT prompts into the autoregressive image generation task enhances the capabilities of large language models in complex image generation tasks without modifying the image generation model structure or the autoregressive raster generation order. This enhances the contextual reasoning ability and stability of the generated results of the autoregressive algorithm's image generation model in complex tasks. In the relevant experimental results of this application embodiment, compared with related autoregressive methods, the performance of image generation in this application embodiment is improved by approximately 20%, significantly improving the quality and consistency of the generated results. It also improves the performance of large language models (LLMs) (examples of image generation models) in image generation.

[0176] In this embodiment, the inference hints added during the image generation process are specially designed image-related inference hints. In this embodiment, an intermediate inference process for predicting image labels is designed so that the image generation model perceives the overall distribution information of the images in the inference hints before generating the target image. For example, the general distribution information of the image dataset in the general hints, or the overall distribution information of images of a specific category in the category hints, thereby enhancing the contextual inference capability of the generation and improving the stability of the generation process by adding inference steps.

[0177] For example, the image generation method in this application still uses the image segmenter architecture and the "next label prediction" method of the classic AR model. Based on this, this application introduces a set of COT (Category-Oriented) prompts, inputting different images of the same category as the target image or randomly sampled images as category prompts into the image generation model. By introducing chain-like thinking, it helps the image generation model understand prior knowledge and assists in generating more logically consistent target images. For example, the inference prompts introduced in this application consist of a label sequence composed of a series of labels, which are concatenated with the labels of the conditional information to generate a new label sequence. When the image generation model predicts the next label, the generated labels, conditional information, and inference prompts guide the generation process. This application references the application of chain-like thinking technology in dynamic programming problems and analyzes the impact of inference prompts on model training. That is, by introducing inference prompts, the image generation model can more quickly approximate the target distribution of the predicted next label compared to the classic AR method, thereby accelerating the convergence of the image generation model.

[0178] Optionally, embodiments of this application provide two different types of inference hints to be incorporated into the training and inference process of the image generation model.

[0179] Furthermore, the category hint information is explained in more detail. During the training phase of the image generation model, a reference image is randomly selected that is different from the target image sample but belongs to the same category as the target image sample. After processing by the image segmenter, it is converted into reference image tags. These reference image tags are then concatenated with the conditional information and input into the image generation model.

[0180] For example, during the training phase of the image generation model, category hints are first placed as tokens after the tokens in the conditional information to predict the tokens in the generated image. The target image sample is first decomposed into tokens by an encoder. Then, using the tokens decomposed from the target image sample as labels, supervised learning is performed by calculating the loss value between the tokens decomposed from the target image sample and the predicted tokens of the generated image. This introduction of chain-like thinking helps the image generation model understand the prior knowledge of the input, thereby generating more logically consistent images.

[0181] Furthermore, in the inference stage of the image generation model, the target category of the generated target image is determined based on the conditional information; a reference image of the same category as the target image is randomly selected, and after word segmentation of the reference image, the token of the reference image is obtained. The token of the reference image is then input into the image generation model as category hint information.

[0182] For example, the process of generating category hints based on conditional information can be accomplished using an image generation model. For instance, by using the conditional information as input, the image generation model can determine the target category of the target image. The model then selects a reference image belonging to the same category as the target image, generates a label sequence for the reference image through its encoder, and outputs this label sequence as the category hint. Alternatively, if the image generation model is a large language model, appropriate hint words can be set to enable it to generate category hints based on the conditional information. Then, based on the input conditional information and the output category hints, the image generation model predicts the labels for each target image, ultimately generating the required label sequence. This generated label sequence is then input into the decoder to obtain the final target image.

[0183] Furthermore, the general prompt information is explained in more detail. From the codebook of the image segmenter, a set of token indices is sampled using a uniform distribution method. The purpose of the uniform distribution method is to select each index with equal probability, thereby improving the generalization of the general prompt information. Then, the sampled index set is concatenated with conditional information and used as input to the image generation model. The codebook is a common token library.

[0184] For example, during the training phase of the image generation model, a token for general prompt information is placed after the token for conditional information to predict the token for the target image. This, along with the predicted tag sequence, is input into the image generation model. During the inference phase, the image generation model can generate general prompt information based on the conditional information. For instance, using the conditional information as input, the model can determine a tag library for multiple categories of the target category indicated by the conditional information and retrieve at least one tag from the tag library as general prompt information. Alternatively, if the image generation model is a large language model, setting appropriate prompt words can enable it to generate general prompt information based on the conditional information.

[0185] It's important to note that category-based hints and general-purpose hints represent two different distribution information used to guide image generation models in generating target images. Category-based hints refer to the distribution corresponding to a specific category (the target category of the target image), and are designed to help the image generation model better understand the distribution of image information corresponding to given conditional information, thereby generating a target image that more clearly represents the conditional information. General-purpose hints, on the other hand, represent a more general image distribution and are designed to encourage the image generation model to generate more diverse and varied images.

[0186] In this embodiment, the image generation model is an autoregressive model. Based on the autoregressive model, a COT (Contextual Reasoning and Opinion) prompt is introduced, and an intermediate inference prompt module for autoregressive image generation is proposed. This module can insert intermediate inference prompts (i.e., inference prompt information) during the image generation process. Through these inference prompts, the autoregressive model can simulate the human reasoning process, performing certain contextual reasoning and global information perception before generating the target image.

[0187] To further illustrate the effectiveness of the methods in the embodiments of this application, related experimental results are also provided in the embodiments of this application.

[0188] This application embodiment conducts experiments on different types of reasoning prompts, including category prompts and general prompts, and compares the effects of using the same or different reasoning prompts during training and reasoning, obtaining the experimental results shown in Table 1.

[0189] In Table 1, "Non" indicates that no inference cues are used; "Mixture" indicates that both category cues and general cues are used. The Fréchet Inception Distance (FID) measures the performance of an image generation model and is an evaluation metric for the quality and diversity of images generated by the model in an image generation task. The Inception Score (IS) primarily assesses the quality of images generated by the model, giving a score by comprehensively considering image sharpness and diversity; a higher IS score indicates better image quality. Precision measures the accuracy or quality of the image generation model's predictions. Recall measures the model's ability to correctly identify positive examples. "↓" indicates that a smaller value indicates better performance, and "↑" indicates a larger value indicates better performance.

[0190] Analysis of the data in Table 1 shows that category-based hints are more effective. For example, when using category-based inference information during the training phase, the FID is 4.34 if category-based inference information is used during the inference phase, and 4.36 if general inference information is used during the inference phase. In other words, category-based inference hints outperform general inference hints.

[0191] Analysis of the data in Table 1 shows that when consistent inference prompts are used in both the training and inference phases of the image generation model, the model performs better than when inconsistent prompts are used. For example, when using category-based inference information during training, the FID is 4.34 if category-based inference information is used in the inference phase, and 4.36 if general inference information is used in the inference phase. This indicates that using category-based inference information in both training and inference phases results in better model performance compared to using category-based inference information in the training phase and general inference information in the inference phase. Conversely, when using general inference information during training, the FID is 4.42 if category-based inference information is used in the inference phase, and 4.39 if general inference information is used in the inference phase. This again demonstrates that using general inference information in both training and inference phases results in better model performance compared to using general inference information in the training phase and category-based inference information in the inference phase.

[0192] Analysis of the data in Table 1 shows that using both category and general cue information simultaneously leads to a partial performance degradation in the image generation model. For example, "Mixture" represents the simultaneous use of both category and general cue information. In the "Mixture" row, the FID is 4.55, which is higher than the FID values ​​in other cases. In the "Mixture" row, the IS is 235.13, indicating that using both category and general cue information simultaneously also leads to a partial performance improvement in the image generation model.

[0193] The method described in this application significantly outperforms baseline methods that do not use inference hints. For example, during both the training and inference phases, regardless of whether consistent or inconsistent inference hints are used, the method described in this application consistently achieves a higher FID score than the baseline method that does not use inference hints. For instance, the baseline method using "Non" (meaning no inference hints) achieves an FID score of 5.46, which is higher than other cases. Therefore, compared to the baseline method, the image generation model exhibits a performance improvement of approximately 20%, demonstrating good robustness and effectiveness.

[0194] Therefore, the results in Table 1 show that, in the embodiments of this application, by introducing reasonable inference prompts, not only is the quality and stability of image generation improved, but also the good adaptability and robustness of the image generation model under different inference prompt configurations are demonstrated.

[0195] Table 1. Results of using different reasoning prompts during training and reasoning processes.

[0196] Figure 7 is a schematic diagram of the ablation study results of the inference hints provided in the embodiments of this application. As shown in Figure 7, the embodiments of this application conducted an ablation study on the length of the inference hints. An ablation study explores the impact of hints of different lengths on model performance. In the ablation study results, when the length of the inference hints is 0, it indicates that no inference hints are set, and the image generation model performs generally. For example, when the number of tags in the inference hints is 80, the FID is 7.4; when the number of tags in the inference hints is 192, the FID is 6.6. Therefore, as the length of the inference hints increases, the FID performance gradually improves. However, when the length of the inference hints approaches 256 tokens, the performance improvement of the image generation model from increasing the length of the inference hints becomes slower.

[0197] In this embodiment of the application, the inference hints were modified at different points in the image generation process of the image generation model for comparative experiments, resulting in Table 2. In Table 2, "Non" indicates that inference hints are not used, assuming this is technical solution 1, also known as the baseline method. "Post-generation hints" in Table 2 indicates that after the image generation model predicts all the markers, all the markers and inference hints are input into the image generation model, which then generates the image, assuming this is technical solution 2. "Blank hints" in Table 2 use the distribution information from a completely black image as inference hints, such as multiple markers from a completely black image, assuming this is technical solution 3. In Table 2, the technical solutions of this embodiment are also referred to as Image Generation with Thoughtful Reasoning (IGTR).

[0198] According to the results in Table 2, in the inference stage of the image generation model, in technical solution 2, when the inference prompts are embedded in different orders and the generated images are generated, the performance of the image generation model in technical solution 2 is comparable to that of the baseline method, technical solution 1, when the inference prompts are placed after image generation. The results of the IGTR solution in this embodiment are superior to those of technical solution 2. For example, in the non-line baseline method, the FID is 8.69 and the IS is 124.44; in technical solution 2 with prompts after generation, the FID is 8.70 and the IS is 122.79; while in the IGTR of this embodiment, the FID is 6.07 and the IS is 174.48. This indicates that the order of inference prompts and image generation has significant value in the image generation process, and is not merely about activating known knowledge through prompts. Therefore, in the IGTR solution of this embodiment, setting inference prompts each time the image generation model predicts the next marker affects the generation result of each marker in the image generation, unlike the ordinary prompt word (technical solution 2) scheme. Technical solution 2 cannot achieve the same influence of inference prompts on the image generation model in IGTR.

[0199] As shown in Table 2, in technical solution 3, the inference hints are multiple blank markers from a completely black image. The performance of the image generation model in technical solution 3 is similar to that of the baseline method in technical solution 1 without inference hints. However, the results of the IGTR method in this embodiment are superior to those of technical solution 3. For example, the baseline method with non-rows has an FID of 8.69 and an IS of 124.44; technical solution 3 with blank hints has an FID of 8.65 and an IS of 123.15; while in the IGTR method of this embodiment, the FID is 6.07 and the IS is 174.48. Technical solution 3 and the IGTR method of this embodiment achieve similar computational resources, indicating that the change in computational resources is not the reason for the success of IGTR, but rather that the distribution information of the image contained in the inference hints plays a key role in improving the performance of the image generation model.

[0200] Table 2 shows the results of setting reasoning prompts at different locations during the image generation process.

[0201] In this application embodiment, a comparative analysis of the IGTR scheme of this application embodiment and different types of models was also studied, and the results are shown in Table 3. The models involved in Table 3 include: models 1-3 corresponding to Generative Adversarial Networks (GAN); models 4-7 corresponding to Diffusion models; models 8-10 corresponding to Masked Autoregressive (Masked AR) models; models 11-21 corresponding to Autoregressive (AR) models; and four models corresponding to the IGTR scheme of this application embodiment.

[0202] As shown in Table 3, compared to GAN, Diffusion, and MaskedAR models, the IGTR solution in this application demonstrates better image generation model scores and superior performance. For example, Model 1 in the GAN model achieves an FID of 6.95 with 112M parameters, while IGTR in this application achieves an FID of 4.39 with 111M parameters. Similarly, Model 6 in the Diffusion model achieves an FID of 3.60 with 400M parameters, while IGTR in this application achieves an FID of 3.23 with 343M parameters. In MaskedAR, Models 8-10 achieve an FID of 4.02 with 227M parameters, while IGTR in this application achieves an FID of 3.23 with 343M parameters.

[0203] As shown in Table 3, compared to the AR model, the IGTR technical solution of this application provides a better image generation model score and superior performance, improving the image generation results in regression-based image generation. For example, in the AR model, Model 12 has an FID of 15.76 with 1.4B parameters; Model 13 has an FID of 5.20 with 1.4B parameters; Model 21 has an FID of 3.09 with 1.4B parameters; and the IGTR of this application has an FID of 2.76 with 1.4B parameters.

[0204] Table 3 shows the results of using the methods of the embodiments of this application in different models.

[0205] Based on the same concept as the foregoing embodiments, this application also provides an image generation apparatus and an image generation model training apparatus.

[0206] Figure 8 shows a schematic diagram of the composition of an image generation apparatus. As shown in Figure 8, this application embodiment provides an image generation apparatus 800, which mainly includes:

[0207] The acquisition module 810 is used to acquire input information.

[0208] The processing module 820 is used to determine inference prompts based on the input information. The inference prompts are used to indicate the distribution information of the image during the image generation process. The processing module 820 is also used to input the input information and the inference prompts into the image generation model to obtain a label sequence of the target image. The label sequence of the target image includes multiple labels corresponding to the target image, and the multiple labels are generated by the image generation model based on the input information and the inference prompts.

[0209] In one possible implementation, the inference prompt information includes category prompt information. Specifically, the processing module 820 described above is used to: determine category prompt information based on the target category indicated by the input information; wherein the target category represents the category of the target image, and the category prompt information represents the distribution information of images within the target category.

[0210] In one possible implementation, the processing module 820 described above is specifically used to: determine a reference image belonging to the target category based on the target category indicated by the input information; generate distribution information of the reference image based on the reference image; and obtain category prompt information based on the distribution information of the reference image.

[0211] In one possible implementation, the processing module 820 described above is specifically used to: perform image word segmentation on the reference image to obtain a tag sequence of the reference image; and obtain the distribution information of the reference image based on the tag sequence of the reference image.

[0212] In one possible implementation, the inference prompt information includes general prompt information. Specifically, the processing module 820 described above is used to: determine general prompt information based on the distribution information of images in multiple categories, whereby the general prompt information represents the distribution information of images in multiple categories, including the target category.

[0213] In one possible implementation, the processing module 820 described above is specifically used to: select at least one tag from a tag library as general prompt information, wherein the tag library includes tag sequences corresponding to images in multiple categories.

[0214] In one possible implementation, the processing module 820 described above is specifically used to: select at least one tag from the tag library as general prompt information using a uniformly distributed sampling method, wherein the uniformly distributed sampling method means that the probability of selecting each tag from the tag library is equal.

[0215] In one possible implementation, the processing module 820 is further configured to: perform image word segmentation on the input information to obtain a labeled sequence of the input information; and perform image word segmentation on the inference prompt information to obtain a labeled sequence of the inference prompt information. Specifically, the processing module 820 is configured to: input the labeled sequence of the input information and the labeled sequence of the inference prompt information into the image generation model.

[0216] In one possible implementation, the input information includes category information, which represents the category of the target image; and / or, the input information includes text information, which represents the descriptive information of the target image.

[0217] In one possible implementation, the target image's label sequence includes n labels, where the i-th label is generated by the image generation model based on input information, inference prompts, and the previously generated (i-1) labels, where n is a positive integer greater than 1 and i is a positive integer greater than 1 and less than or equal to n.

[0218] Figure 9 shows a schematic diagram of the composition of an image generation device. As shown in Figure 8, this application embodiment provides a training device 900 for an image generation model, mainly including:

[0219] The training dataset construction module 910 is used to construct the training dataset. The training dataset includes target image samples, ground truth values ​​of the labeled sequences of the target image samples, conditional samples of the target image samples, and inference hint samples of the target image samples. The ground truth values ​​of the labeled sequences are obtained by image word segmentation of the target image samples. The conditional samples represent the relevant conditions of the generated images, and the inference hint samples are used to indicate the distribution information of the generated images.

[0220] The model training module 920 is used to input conditional samples and inference cue samples into the image generation model to obtain the predicted values ​​of the label sequence of the generated image, the predicted values ​​of the label sequence including multiple labels generated by the image generation model based on the conditional samples and inference cue samples; and to determine the loss value based on the ground truth value of the label sequence and the predicted value of the label sequence; and to adjust the parameters of the image generation model based on the loss value.

[0221] In one possible implementation, the inference hint samples include category hint samples. Specifically, the training dataset construction module 910 described above is used to: determine category hint samples based on the category of the target image sample, whereby the category hint samples represent the distribution information of images within the category of the target image sample.

[0222] In one possible implementation, the training dataset construction module 910 described above is specifically used to: determine reference image samples based on conditional samples, wherein the reference image samples are different images of the same category as the target image samples; perform image word segmentation on the reference image samples to obtain a label sequence of the reference image samples; and use the label sequence of the reference image samples as category cue samples.

[0223] In one possible implementation, the inference hint samples include general hint samples. The training dataset construction module 910 described above is specifically used to: determine general hint samples based on the distribution information of images in multiple categories, whereby the general hint samples represent the distribution information of images in multiple categories, including the category of the target image sample.

[0224] In one possible implementation, the training dataset construction module 910 described above is specifically used to: construct or acquire image datasets of multiple categories; generate or acquire label sequences of images in the image dataset; and select at least one label from the label sequences of images in the image dataset as a general cue sample.

[0225] The software and hardware implementation of the image generation device 800 shown in Figure 8 and the image generation model training device 900 shown in Figure 9 (hereinafter referred to as: the device of this application embodiment) will be further explained below.

[0226] As an example of a software functional unit, a module can include code running on a computing instance. A computing instance can include at least one of a physical host (computing device), a virtual machine, or a container. Furthermore, the aforementioned computing instance can be one or more. For example, a module can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code can be distributed within the same availability zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region can include multiple AZs.

[0227] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0228] As an example of a hardware functional unit, a module may include at least one computing device, such as a server. Alternatively, a module may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The aforementioned PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0229] The multiple computing devices included in the module can be distributed within the same region or in different regions. Similarly, the multiple computing devices included in the module can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the module can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0230] It should be noted that, in other embodiments, the apparatus of this application embodiment additionally provides one or more modules for performing any of the steps included in the above implementation. The steps implemented by one or more modules in the apparatus of this application embodiment can be specified as needed, and more or fewer modules can be obtained than in the embodiments of this application to implement different steps in the above method, thereby realizing all the functions of the apparatus of this application embodiment.

[0231] This application also provides a computing device 1000. As shown in FIG10, the computing device 1000 includes: a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, the memory 1006, and the communication interface 1008 communicate with each other via the bus 1002. The computing device 1000 can be a server, such as a central server, an edge server, or a local server in a local data center, or it can be an electronic device such as a desktop computer, a laptop computer, or a smartphone. It should be understood that this application does not limit the number of processors and memories in the computing device 1000.

[0232] Bus 1002 can be a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL), a Cache Coherent Interconnect for Accelerators (CCIX), etc. The Unified Bus is also known as the Lingqu Bus. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 10 uses only one line, but this does not imply that there is only one bus or one type of bus. Bus 1004 can include pathways for transmitting information between various components of the computing device 1000 (e.g., memory 1006, processor 1004, communication interface 1008).

[0233] The processor 1004 may include any one or more of the following computing devices: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP) or digital signal processor (DSP), ASIC, FPGA, CPLD, NPU, SoC, offload card, accelerator card, etc.

[0234] The memory 1006 may include volatile memory, such as random access memory (RAM). The processor 1004 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). Furthermore, the memory 1006 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0235] It is worth noting that the same type of storage medium can be configured in the same computing device to realize the function of memory 1006, or two or more types of storage media can be configured to realize the function of memory 1006. This application does not limit this.

[0236] The memory 1006 stores executable program code, and the processor 1004 executes the executable program code to implement the functions of the apparatus in the embodiments of this application, thereby implementing the methods described in the above embodiments. That is, the memory 1006 stores instructions for executing the methods described in the above embodiments.

[0237] Alternatively, the memory 1006 may store executable code, which the processor 1004 executes to implement the functions of the apparatus in the embodiments of this application, thereby implementing the methods described in the above embodiments. That is, the memory 1006 stores instructions for executing the methods described in the above embodiments.

[0238] The communication interface 1008 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1000 and other devices or communication networks.

[0239] As one possible implementation, the computing device 1000 may also include a chip system, which includes a processor and a power supply circuit. The power supply circuit supplies power to the processor, and the processor executes the operation steps corresponding to the methods in the embodiments of this application. For simplicity, further details are omitted here. The processor can be implemented using a GPU, or it can be implemented using computing devices or AI chips such as a DPU, NPU, XPU, SoC, offloading card, or accelerator card.

[0240] As one possible implementation, the computing device 1000 may include various types of processors 1004, that is, the computing device 1000 is a heterogeneous device. For example, the computing device 1000 includes a CPU and a GPU, and at least one of the processors 1004 can execute the operation steps corresponding to the method of the embodiments of this application. For the sake of brevity, further details are omitted here.

[0241] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be an electronic device such as a desktop computer, a laptop computer, or a smartphone.

[0242] As shown in Figure 11, the computing device cluster includes at least one computing device 1000. The memory 1006 of one or more computing devices 1000 in the computing device cluster may store the same instructions for performing the methods described in the above embodiments.

[0243] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store partial instructions for executing the methods described in the above embodiments. In other words, a combination of one or more computing devices 1000 can jointly execute instructions for executing the methods described in the above embodiments.

[0244] It should be noted that the memory 1006 in different computing devices 1000 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the apparatus in this application embodiment. That is, the instructions stored in the memory 1006 of different computing devices 1000 can implement the functions of one or more modules of the apparatus in this application embodiment.

[0245] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 12 illustrates one possible implementation. As shown in Figure 12, two computing devices 1000A and 1000B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 1006 in computing device 1000A stores instructions for the functions of one or more modules of the apparatus according to this application embodiment. Simultaneously, the memory 1006 in computing device 1000B stores instructions for the functions of another one or more modules of the apparatus according to this application embodiment.

[0246] It should be understood that the functions of computing device 1000A shown in Figure 12 can also be performed by multiple computing devices 1000. Similarly, the functions of computing device 1000B can also be performed by multiple computing devices 1000.

[0247] This application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similarly referred to the connection method of the computing device cluster described in Figures 11 and 12. The difference is that the memory 1006 of one or more computing devices 1000 in this computing device cluster can store the same instructions for executing the methods in the above embodiments.

[0248] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store partial instructions for executing the aforementioned data processing method. In other words, a combination of one or more computing devices 1000 can jointly execute the instructions for performing the aforementioned method.

[0249] Based on the methods in the above embodiments, this application provides a computer-readable storage medium including computer program instructions. When executed by a cluster of computing devices including at least one computing device, the computer program instructions cause the cluster of computing devices to perform the methods in the above embodiments. Exemplarily, the computer-readable storage medium can be any available medium capable of being stored in the cluster of computing devices or a data storage device such as a data center containing one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives).

[0250] Based on the methods in the above embodiments, this application provides a computer program product containing instructions. The computer program product may be software or program products containing instructions, capable of running on a computing device or stored on any available medium. When the instructions are executed by a cluster of computing devices containing at least one computing device, at least one computing device in the cluster of computing devices performs the methods in the above embodiments.

[0251] It is understood that the processor in the embodiments of this application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor may be a microprocessor or any conventional processor.

[0252] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0253] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0254] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0255] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. An image generation method, characterized in that, include: Obtain input information; Based on the input information, inference prompt information is determined, which is used to prompt the distribution information of the image during the image generation process; The input information and the inference prompt information are input into the image generation model to obtain a label sequence of the target image. The label sequence of the target image includes multiple labels corresponding to the target image, and the multiple labels are generated by the image generation model based on the input information and the inference prompt information. The target image is generated based on the tag sequence of the target image.

2. The method according to claim 1, characterized in that, The reasoning prompts include category prompts; The step of determining the reasoning prompt information based on the input information includes: Based on the target category indicated by the input information, the category prompt information is determined; wherein, the target category represents the category of the target image, and the category prompt information represents the distribution information of the images in the target category.

3. The method according to claim 2, characterized in that, The step of determining the category prompt information based on the target category indicated by the input information includes: Based on the target category indicated by the input information, determine the reference image belonging to the target category; Based on the reference image, generate the distribution information of the reference image; The category prompt information is obtained based on the distribution information of the reference image.

4. The method according to claim 3, characterized in that, The step of generating the distribution information of the reference image based on the reference image includes: The reference image is subjected to image word segmentation to obtain the tag sequence of the reference image; The distribution information of the reference image is obtained based on the label sequence of the reference image.

5. The method according to any one of claims 1-4, characterized in that, The reasoning prompts include general prompts; The step of determining the reasoning prompt information based on the input information includes: The general prompt information is determined based on the distribution information of images in multiple categories. The general prompt information represents the distribution information of images in the multiple categories, including the target category indicated by the input information.

6. The method according to claim 5, characterized in that, The step of determining the general prompt information based on the distribution information of images in multiple categories includes: At least one tag is selected from the tag library as the general prompt information, and the tag library includes the tag sequence corresponding to the images in the multiple categories.

7. The method according to claim 6, characterized in that, Selecting at least one tag from the tag library as the general prompt information includes: Using a uniformly distributed sampling method, at least one tag is selected from the tag library as the general prompt information, wherein the uniformly distributed sampling method means that the probability of selecting each tag from the tag library is equal.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: The input information is processed by image word segmentation to obtain a labeled sequence of the input information; The inference prompt information is processed by image word segmentation to obtain the labeled sequence of the inference prompt information; The step of inputting the input information and the reasoning prompt information into the image generation model includes: The labeled sequence of the input information and the labeled sequence of the reasoning prompt information are input into the image generation model.

9. The method according to any one of claims 1-8, characterized in that, The input information includes category information, which represents the category of the target image; and / or, the input information includes text information, which represents the descriptive information of the target image.

10. The method according to any one of claims 1-9, characterized in that, The target image's label sequence includes n labels. The i-th label among the n labels is generated by the image generation model based on the input information, the inference prompt information, and the previously generated i-1 labels. n is a positive integer greater than 1, and i is a positive integer greater than 1 and less than or equal to n.

11. A training method for an image generation model, characterized in that, include: A training dataset is constructed, which includes target image samples, ground truth values ​​of the labeled sequences of the target image samples, conditional samples of the target image samples, and inference hint samples of the target image samples. The ground truth values ​​of the labeled sequences are obtained by performing image word segmentation on the target image samples. The conditional samples represent the relevant conditions of the generated images, and the inference hint samples are used to indicate the distribution information of the generated images. The conditional samples and the inference hints samples are input into the image generation model to obtain the predicted value of the label sequence of the generated image. The predicted value of the label sequence includes multiple labels, which are generated by the image generation model based on the conditional samples and the inference hints samples. The loss value is determined based on the true value of the labeled sequence and the predicted value of the labeled sequence; The parameters of the image generation model are adjusted based on the loss value.

12. The method according to claim 11, characterized in that, The reasoning hint samples include category hint samples; The process of constructing the category suggestion sample includes: Based on the category of the target image sample, a category hint sample is determined, wherein the category hint sample represents the distribution information of the images in the category of the target image sample.

13. The method according to any one of claims 11 or 12, characterized in that, The reasoning hint samples include general hint samples; The process of constructing the general prompt sample includes: Based on the category of the target image sample, multiple categories are determined, including the category of the target image sample; Based on the distribution information of images in the multiple categories, a general prompt sample is determined, wherein the general prompt sample represents the distribution information of images in the multiple categories.

14. An image generation apparatus, characterized in that, include: The acquisition module is used to acquire input information; The processing module is used to determine inference prompt information based on the input information, the inference prompt information being used to prompt the distribution information of the image during the image generation process; Furthermore, the input information and the inference prompt information are input into the image generation model to obtain a label sequence of the target image. The label sequence of the target image includes multiple labels corresponding to the target image, and the multiple labels are generated by the image generation model based on the input information and the inference prompt information. And, the target image is generated based on the tag sequence of the target image.

15. The apparatus according to claim 14, characterized in that, The reasoning hint information includes category hint information; the processing module is specifically used for: Based on the target category indicated by the input information, the category prompt information is determined; wherein, the target category represents the category of the target image, and the category prompt information represents the distribution information of the images in the target category.

16. The apparatus according to claim 14 or 15, characterized in that, The reasoning hint information includes general hint information; the processing module is specifically used for: The general prompt information is determined based on the distribution information of images in multiple categories. The general prompt information represents the distribution information of images in the multiple categories, including the target category indicated by the input information.

17. A training device for an image generation model, characterized in that, include: A training dataset construction module is used to construct a training dataset, which includes target image samples, ground truth values ​​of the labeled sequences of the target image samples, conditional samples of the target image samples, and inference hint samples of the target image samples. The ground truth values ​​of the labeled sequences are obtained by performing image word segmentation on the target image samples. The conditional samples represent the relevant conditions of the generated images, and the inference hint samples are used to indicate the distribution information of the generated images. The model training module is used to input the conditional samples and the inference hint samples into the image generation model to obtain the predicted value of the label sequence of the generated image. The predicted value of the label sequence includes multiple labels, which are generated by the image generation model based on the conditional samples and the inference hint samples. Furthermore, the loss value is determined based on the true value of the labeled sequence and the predicted value of the labeled sequence; Furthermore, the parameters of the image generation model are adjusted based on the loss value.

18. A chip system, characterized in that, The chip system includes a processor and a power supply circuit, the power supply circuit being used to supply power to the processor, the processor being used to perform the method as described in any one of claims 1-13.

19. A computing device, characterized in that, The computing device includes a processor and memory; The processor is configured to execute instructions stored in the memory to cause the computing device to perform the method as described in any one of claims 1-13.

20. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-13.

21. A computer-readable storage medium, characterized in that, The method includes computer program instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any one of claims 1-13, wherein the cluster of computing devices includes at least one computing device.

22. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1-13, wherein the computing device cluster includes at least one computing device.

Images