Image label generation method, related apparatus and medium

Abstract

The present disclosure provides an image label generation method, a related apparatus and a medium. The method comprises: acquiring a first image and first image description text for describing the first image; extracting keywords from the first image description text to serve as keyword labels corresponding to the first image; by means of a multi-label classification model, determining a target category label corresponding to the first image, wherein the target category label is used for representing an image category of the first image, the target category label comprises at least one of a plurality of preset candidate category labels, the multi-label classification model comprises a plurality of label sub-models respectively corresponding to the plurality of candidate category labels, and each label sub-model is used for determining a matching degree between the corresponding candidate category label and the first image; and determining the keyword labels and the target category label as associated labels of the first image. The present disclosure can generate associated labels with high semantic relevance with images, and can be widely applied to scenarios such as image retrieval and image recommendation.

Description

Image tag generation methods, related devices and media

[0001] This application claims priority to Chinese Patent Application No. 2025101920220, filed on February 20, 2025, entitled “Image Tag Generation Method, Related Apparatus and Medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of artificial intelligence, and in particular to an image label generation method, related apparatus and medium. Background Technology

[0003] Image tags are text labels used to aid in understanding the content of an image. They reflect the content contained in the image, and by pre-labeling images with appropriate image tags, subsequent image retrieval can be performed based on these tags. When generating image tags, in addition to the original image tags, associative tags are often added to accommodate situations where different users may use different texts to describe similar images.

[0004] However, in related technologies, associative tags are often generated based on statistical correlation, that is, words that co-occur frequently with the original tags of the image are often used as associative tags for the image. This makes it possible for associative tags to have no direct semantic connection with the image, and the semantic correlation between associative tags and the image is low, which cannot accurately reflect the image content. Summary of the Invention

[0005] This disclosure provides an image tag generation method, related apparatus, and medium that enables associative tags to have a higher semantic correlation with images and to more accurately reflect image content.

[0006] According to one aspect of this disclosure, an image tag generation method is provided, performed by an electronic device, comprising:

[0007] Acquire a first image and a first image description text used to describe the first image;

[0008] Keywords are extracted from the description text of the first image and used as keyword tags corresponding to the first image;

[0009] A target category label corresponding to the first image is determined using a multi-label classification model. The target category label is used to characterize the image category of the first image. The target category label includes at least one of a plurality of preset candidate category labels. The multi-label classification model includes a plurality of label sub-models corresponding to the plurality of candidate category labels. Each label sub-model is used to determine the matching degree between the corresponding candidate category label and the first image. The target category label is determined based on the matching degree between each candidate category label and the first image.

[0010] The keyword tags and the target category tags are determined as the associated tags for the first image.

[0011] According to one aspect of this disclosure, an image tag generation apparatus is provided, the apparatus comprising:

[0012] The acquisition unit is used to acquire a first image and a first image description text for describing the first image;

[0013] The keyword extraction unit is used to extract keywords from the first image description text and use them as keyword tags corresponding to the first image.

[0014] A category label generation unit is used to determine a target category label corresponding to the first image through a multi-label classification model. The target category label is used to characterize the image category of the first image. The target category label includes at least one of a plurality of preset candidate category labels. The multi-label classification model includes a plurality of label sub-models corresponding to the plurality of candidate category labels. Each label sub-model is used to determine the matching degree between the corresponding candidate category label and the first image. The target category label is determined based on the matching degree between each candidate category label and the first image.

[0015] A determining unit is used to determine the keyword tag and the target category tag as the associated tags of the first image.

[0016] According to one aspect of this disclosure, an electronic device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the image tag generation method as described above.

[0017] According to one aspect of this disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program that, when executed by a processor, implements the image tag generation method as described above.

[0018] According to one aspect of this disclosure, a computer program product is provided, comprising a computer program that is read and executed by a processor of a computer device, causing the computer device to perform the image tag generation method as described above.

[0019] The image tag generation method, related apparatus, and medium proposed in this disclosure extract keywords from the first image description text used to describe the first image to obtain keyword tags for the first image, ensuring a direct semantic connection between the keyword tags and the first image. Simultaneously, a multi-label classification model calculates the matching degree between the first image and each candidate category tag, enabling the determination of the semantic matching degree between the candidate category tags and the first image. At least one candidate category tag with a high semantic matching degree is identified as the target category tag corresponding to the first image, ensuring a semantic association between the target category tag and the first image. The keyword tags and the target category tag are then identified as the associated tags corresponding to the first image. Furthermore, since the multi-label classification model includes multiple tag sub-models corresponding to each candidate category tag, each tag sub-model can independently determine the matching degree for its corresponding candidate category tag, making the matching degree more accurate. Thus, while ensuring the richness of the associated tags for the first image, it can also ensure that these associated tags are semantically related to the first image, thereby more accurately reflecting the image content.

[0020] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objectives and other advantages of this disclosure may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0021] The accompanying drawings are provided to further understand the technical solutions of this disclosure and constitute a part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure.

[0022] Figure 1 is a system architecture diagram of the image tag generation method according to an embodiment of the present disclosure.

[0023] Figure 2 is a flowchart illustrating an image tag generation method provided in an embodiment of this disclosure;

[0024] Figure 3 is a schematic diagram of an image tag generation method provided in an embodiment of this disclosure;

[0025] Figure 4 is a schematic diagram of the label determined based on point mutual information in related technologies;

[0026] Figure 5 is a schematic diagram of the process of updating the multi-label classification model when new candidate category labels are added, as provided in the embodiments of this disclosure.

[0027] Figure 6 is a schematic diagram of the training process of the first image encoder and the first text encoder provided in the embodiments of this disclosure;

[0028] Figure 7 is a schematic diagram of the image retrieval process provided in an embodiment of this disclosure;

[0029] Figure 8 is a schematic diagram of the interface during the image retrieval process provided in an embodiment of this disclosure;

[0030] Figure 9 is a schematic diagram of the display interface of the image retrieval results provided in an embodiment of this disclosure;

[0031] Figure 10 is a block diagram of an image tag generation apparatus according to an embodiment of the present disclosure;

[0032] Figure 11 is a terminal structure diagram according to an embodiment of the present disclosure;

[0033] Figure 12 is a server structure diagram according to an embodiment of the present disclosure. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.

[0035] Before providing a further detailed description of the embodiments of this disclosure, the terms and concepts used in these embodiments are explained, and they are subject to the following interpretations:

[0036] Related tags (RT), also known as relevant tags or recommended tags, are image tags used to characterize image content. They are text tags used in text-based image retrieval. By pre-generating related tags for images and storing them in association, when a user needs to perform a text-based image retrieval, the system can parse the user's input text, generate corresponding tags, and match these generated tags with the pre-stored related tags to retrieve the desired image.

[0037] Image tags are text labels used to help indicate the content of an image. They can reflect the content contained in the image, and by pre-labeling images with appropriate image tags, subsequent image retrieval can be performed based on these tags. When generating image tags, in addition to the original image tags, associative tags are often added to accommodate situations where different users may use different texts to describe similar images. However, in related technologies, associative tags are often generated based on statistical correlation, meaning that words that co-occur frequently with the original image tags are often used as associative tags. This results in associative tags often lacking a direct semantic connection with the image, leading to low semantic correlation and failing to accurately reflect the image content.

[0038] Based on this, this disclosure proposes an image tag generation method that can ensure a high semantic correlation between the associated tags and the image, and can accurately reflect the image content.

[0039] Figure 1 is a system architecture diagram of the image tag generation method according to an embodiment of the present disclosure. It includes: an object terminal 110, an Internet 120, a gateway 130, and a server 140.

[0040] The object terminal 110 is a device used by the object to input the first image and the first image description text. It includes various forms such as desktop computers, laptops, PDAs (Personal Digital Assistants), mobile phones, in-vehicle terminals, home theater terminals, and dedicated terminals. Furthermore, it can be a single device or a collection of multiple devices. For example, multiple devices can be connected via a local area network, sharing a single display device to work collaboratively, forming a single terminal. The object terminal 110 can also communicate with the Internet 120 via wired or wireless means to exchange data.

[0041] Gateway 130, also known as an internetwork connector or protocol converter, is a computer system or device that enables network interconnection at the transport layer and acts as a translator. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateway 130 also provides filtering and security functions. Messages sent from target terminal 110 to server 140 are forwarded to the corresponding server 140 via gateway 130. Messages sent from server 140 to target terminal 110 are also forwarded to the corresponding target terminal 110 via gateway 130.

[0042] Server 140 refers to a computer system capable of providing image tag generation services to target terminal 110. Compared to target terminal 110, server 140 has higher requirements in terms of stability, security, and performance. Server 140 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines). Server 140 can also communicate with the Internet 120 via wired or wireless means to exchange data.

[0043] For example, the target terminal 110 sends a first image and a first image description text to the server 140. The server 140 can obtain the first image and the first image description text, and extract keywords from the first image description text as keyword tags corresponding to the first image. The server 140 can call a multi-label classification model, and through multiple label sub-models included in the multi-label classification model, determine the matching degree between the first image and multiple preset candidate category labels respectively, and determine the target category label corresponding to the first image from the multiple preset candidate category labels. The server 140 then determines the target category label and keyword label as the associated label corresponding to the first image, and sends the associated label corresponding to the first image to the target terminal 110 so as to display the associated label corresponding to the first image in the interface of the target terminal 110.

[0044] In addition, the server 140 may not send the associated tag corresponding to the first image to the target terminal 110. Instead, it may perform subsequent processing directly on the server 140. For example, the server 140 may directly associate and store the first image and the corresponding associated tag in the image database preset by the server 140. After receiving the image retrieval request sent by the target terminal 110, the server 140 may retrieve the corresponding target image from multiple first images in the image database based on the target associated tag indicated by the image retrieval request and return the target image to the target terminal 110.

[0045] As can be seen, compared with the related technology that determines the associated tags of the first image based on the co-occurrence frequency between tags, this disclosure extracts keyword tags used to describe the first image from the description text of the first image, and determines the semantic matching degree between the first image and each preset candidate category tag through a tag classification model. In this way, several candidate category tags with a high semantic matching degree with the first image are determined from the preset multiple candidate category tags as target category tags, and these are used as associated tags corresponding to the first image. This can make the generated associated tags have a higher semantic correlation with the first image.

[0046] The image tag generation method disclosed herein can be applied to various scenarios, such as image retrieval, dynamic image matching, and emoji recommendation.

[0047] Referring to Figure 2, which is an optional flowchart of an image tag generation method provided in this embodiment of the present disclosure, the image tag generation method can be executed by an electronic device, for example, it can be executed independently by a server, or independently by a terminal, or it can be executed collaboratively by a terminal and a server, with the terminal and the server each completing a part of the process. The image tag generation method may include, but is not limited to, the following steps:

[0048] Step 201: Obtain the first image and the first image description text used to describe the first image.

[0049] The first image is the image that needs to be stored in a preset database so that it can be used as a search result in subsequent image retrieval. Specifically, the first image can be an image uploaded by the user through the terminal, or it can be an image obtained from various Internet platforms and stored in the corresponding database. This embodiment does not limit this.

[0050] The first image description text is text used to describe the content of the first image. It can be used to describe image elements such as people and objects contained in the first image, as well as attributes such as the color, shape, and positional relationship of each image element. For example, if the first image is an anime character's head, the corresponding first image description text can be text used to describe the facial features of the anime character, such as hair color, face shape, and expression. Specifically, the first image description text can be text uploaded synchronously when the user uploads the first image through the terminal. For example, in one possible embodiment, an image upload box and a text upload box can be set up simultaneously in the application interface. When an upload request is received, the image in the image upload box is obtained as the first image, and the text entered in the text upload box is obtained as the first image description text. Alternatively, the first image description text can be text obtained from the page where the first image is located when the first image is obtained from the Internet platform. Or, the first image description text can also be text entered by the user based on the image of the first image after it has been obtained, describing the content in the first image. In this embodiment, the method of obtaining the first image description text is not limited.

[0051] Step 202: Extract keywords from the description text of the first image and use them as keyword tags corresponding to the first image.

[0052] Keyword tags are lexical tags used to characterize the content of the first image. They are composed of words contained in the descriptive text of the first image. Each keyword tag characterizes a specific image element or attribute of an image element in the first image. For example, if the first image is an image of an anime character and the descriptive text includes "black hair," then "black hair" can be extracted as a keyword tag for that first image. It is understood that there can be one or more keyword tags, and the specific number can be determined based on the actual descriptive text of the first image; this embodiment does not impose any limitation on this.

[0053] Specifically, in this embodiment, keywords can be extracted from the first image description text using natural language processing methods to obtain keyword tags corresponding to the first image. For example, referring to FIG3, the first image description text is input into a large language model to extract keywords from the first image description text using the semantic understanding capability of the large language model, and these keywords are used as keyword tags corresponding to the first image.

[0054] In one possible implementation, step 202, extracting keywords from the first image description text, includes: filling the first image description text into a preset prompt template to obtain the first input text; the preset prompt template includes at least one logical analysis prompt word, which is used to characterize the analysis dimension when the first language model performs semantic analysis on the first image description text; and inputting the first input text into the first language model so that the first language model performs semantic analysis on the first image description text based on the analysis dimension indicated by the logical analysis prompt word to obtain keywords.

[0055] Based on this, by filling the first image description text into a preset prompt template containing at least one logical analysis prompt word, and inputting the filling result into the first large language model, the first large language model is instructed by the logical analysis prompt word to perform semantic analysis on the first image description text according to the corresponding analysis dimension. In this way, the semantic understanding capability of the large language model is used to extract keywords from the first image description text. Thus, the extracted keywords can accurately represent the main content in the first image description text, ensuring a high semantic correlation between the keyword tags and the first image.

[0056] Specifically, the preset prompt template not only includes task prompts that instruct the first language model to perform semantic analysis on the first image description text in order to extract keywords from the first image description text, but also includes at least one logical analysis prompt. These logical analysis prompts are used to instruct the first language model on the analysis dimensions when analyzing the first image description text, so that the first language model analyzes the first image description text from a specific dimension. For example, a certain logical analysis prompt can be used to instruct the first language model to analyze the scene type described in the first image description text and then extract words that match the scene type from the first image description text as keywords.

[0057] Specifically, the preset prompt template contains fields for filling in the first image description text. For example, the preset prompt template could be "Please analyze the following text <first image description text>, <logic analysis prompt word 1>, <logic analysis prompt word 2>...". After filling the first image description text into the corresponding fields in the preset prompt template, the first input text is obtained. Then, the first input text is input into the first large language model, which performs semantic analysis on the first image description text based on the analysis dimensions indicated by each logical analysis prompt word, thereby extracting keyword tags corresponding to the first image from the first image description text.

[0058] In one possible embodiment, when the preset prompt template includes multiple logical analysis prompts, the analysis dimensions indicated by the multiple logical analysis prompts can be interconnected and progressively advanced. That is, each logical analysis prompt is used sequentially to instruct the first language model to perform semantic analysis on the first image description text according to the next analysis dimension based on the analysis dimension indicated by the previous logical analysis prompt. At this time, the multiple logical analysis prompts constitute a thought chain to instruct the first language model how to progressively analyze the first image description text to extract keywords. Through the thought chain formed by the multiple logical analysis prompts, the semantic analysis task of the first language model on the first image description text can be decomposed into multiple progressively advanced reasoning sub-tasks, so that the first language model can progressively complete the corresponding reasoning sub-tasks, obtain the corresponding intermediate reasoning results, and combine the corresponding intermediate reasoning results to analyze the first image description text based on the analysis dimension indicated by the next logical analysis prompt.

[0059] For example, the first logical analysis prompt can be used to instruct the first language model to analyze the scene type described by the first image description text; the second logical analysis prompt can be used to prompt the first language model to determine which image entities described in the first image description text have higher importance under the corresponding scene type; and the third logical analysis prompt can be used to instruct the first language model to extract attributes from the first image description text to describe these image entities with higher importance.

[0060] In one embodiment, the logical analysis prompts also include logical analysis verification prompts for instructing the first language model to perform logical analysis of the keyword extraction results. Through these logical analysis verification prompts, the first language model can perform semantic analysis on the first image description text according to each logical analysis prompt, extract keywords from the first image description text, and then verify whether the keyword extraction process conforms to the analysis dimensions indicated by each logical analysis prompt, and whether the extracted keywords match the intermediate inference results generated by the semantic analysis based on each logical analysis prompt.

[0061] For example, in the preset prompt template, the analysis logic verification prompts can be "Information coverage: detect whether the analysis results cover the main content of the text; Logical relationship: check whether the intermediate inference results obtained according to the analysis dimensions indicated by each logical analysis prompt are related; Scene matching degree: detect whether the extracted keywords match the scene category." Based on these analysis logic verification prompts, the first language model can verify whether the extracted key information covers the main content of the first image description text, whether the step-by-step inference process based on each logical analysis prompt is correct, and whether the extracted keywords match the intermediate inference results obtained from semantic analysis based on each logical analysis prompt, thereby further ensuring the accuracy of the extracted keywords.

[0062] In one possible implementation, the first input text may also include a preset example. That is, the preset example, along with the first image description text and a preset prompt template, is input into the first language model. The preset example includes example text to be analyzed and example analysis results. The example text to be analyzed is text used to describe the content of an image. The example analysis results are the intermediate inference results obtained by step-by-step analysis of the example text according to the thought chain indicated by the preset prompt template, as well as the keywords extracted from the example text. Because the preset example demonstrates an example of analyzing text according to the thought chain indicated by the preset prompt template, by inputting the preset example as additional contextual information along with the first input text into the first language model, the first language model can further learn how to analyze text according to the preset prompt template based on the preset example before performing semantic analysis on the first image description text, thereby further improving the accuracy of the first language model in performing semantic analysis on the first image description text.

[0063] In related technologies, keyword extraction from image description text is often based on term frequency. For example, by calculating the frequency of each word in the image description text, words with higher or lower frequencies are used as image keyword tags. For instance, in the term frequency–inverse document frequency (TF-IDF) method, the sum of the frequencies of each word in all texts (i.e., the multiple first image description texts corresponding to multiple first images) is recorded, and words with high frequencies in the currently processed text and low sums of frequencies in other texts are identified as keywords. This leads to a tendency in related technologies to use words with low semantic relevance to the image but whose frequency meets preset rules as keyword tags corresponding to the image, resulting in extracted keyword tags that may have low semantic relevance to the corresponding image.

[0064] Specifically, referring to Figure 4, which shows the result of extracting keywords from text using point mutual information as an indicator, it is easy to see that when extracting keywords based on point mutual information, some words unrelated to the image are often extracted as image tags. For example, for an image with a portrait as its theme, words such as "auspicious" and "passer" that are not actually related to the portrait are extracted as corresponding keyword tags; for an image with anime as its theme, words such as "confidence" that are not actually related to anime are extracted as corresponding keyword tags; for an image with silver hair as its theme, words such as "white fox headdress" and "half-body portrait" that are not actually related to silver hair are extracted as corresponding keyword tags; for an image with a watercolor-like theme, words such as "confidence" that are not actually related to watercolor-like images are extracted as corresponding keyword tags; and for an image with a corporate culture wall as its theme, words such as "scenery" and "creative text" that are not actually related to corporate culture walls are extracted as corresponding keyword tags.

[0065] In this embodiment, the first image description text is filled into a preset prompt template and then input into a first language model for semantic analysis. The first language model performs semantic analysis on the first image description text step by step according to the logical analysis prompt words in the preset prompt template, and extracts keywords from the first image description text based on the semantic analysis results, ensuring that there is a direct semantic connection between the extracted keyword tags and the first image.

[0066] In one possible implementation, to enable the first language model to extract corresponding keywords more accurately from the first image description text, the first language model may be trained with additional fine-tuning. Specifically, the training process of the first language model includes: acquiring sample text and decomposing the sample text into at least one candidate sample keyword; inputting the sample text and candidate sample keywords into a second language model to determine the sample keyword corresponding to the sample text from the candidate sample keywords; inputting the sample text and a preset prompt template into the first language model to obtain the predicted sample keywords output by the first language model, and optimizing the parameters of the first language model based on the difference between the predicted sample keywords and the sample keywords.

[0067] Based on this, by first decomposing the sample text into multiple candidate sample keywords, and then inputting the sample text and the candidate sample keywords into the second large language model, using the semantic understanding ability of the second large language model to first perform a keyword recognition task with lower difficulty to identify whether each candidate sample keyword is a sample keyword corresponding to the sample text, thereby obtaining the sample keywords corresponding to the sample text. In this way, the sample text and sample keywords required for training the first large language model to perform the keyword extraction task can be automatically generated based on the sample text, and the training data required for training the first large language model can be obtained. After that, the first large language model is trained using the sample text and the sample keywords identified by the second large language model, so that the first large language model can learn how to extract keywords from text more accurately.

[0068] The sample text can be any text with real meaning. It can be a piece of text input by an object or text obtained from various Internet platforms. In this regard, no restrictions are imposed in this embodiment. It can be understood that the sample text contains at least one vocabulary with practical meaning, such as nouns, verbs, quantifiers, etc., to ensure that at least one sample keyword related to the semantics of the sample text can be extracted from the sample text. In this embodiment, the sample text is used to train the first large language model to perform keyword extraction in the subsequent process, and it can be text that does not correspond to any image. It can be understood that the data required for fine-tuning the first large language model is often multiple, and correspondingly, the number of sample texts can be multiple.

[0069] The candidate sample keywords are each vocabulary with real semantics extracted from the corresponding sample text. It can be understood that in this embodiment, the sample text is decomposed to obtain each candidate sample keyword, and each obtained candidate sample keyword includes every vocabulary in the sample text without missing any vocabulary in the sample text.

[0070] Specifically, after obtaining the sample text, the sample text can be tokenized at the word granularity. For example, the Natural Language Toolkit (NTLK) tool can be used to tokenize the sample text, or the sample text can be tokenized based on certain preset rules. No limitations are imposed in this embodiment. After tokenizing the sample text at the word granularity, multiple candidate sample keywords in the sample text can be obtained. For example, if the sample text is "A cat is running", after tokenizing this sample text using a tokenization tool, the four words "A", "cat", "is", and "running" can be obtained as candidate sample keywords.

[0071] The second large language model can be a pre-trained large language model that can be used to handle various text reasoning tasks. The second large language model and the first large language model can be two large language models built on different neural network structures; or, the second large language model and the first large language model can also be two large language models based on the same neural network structure; of course, in one possible embodiment, the second large language model and the first large language model can also be the same large language model, which is not limited in this embodiment.

[0072] Specifically, after segmenting the sample text, the sample text and candidate sample keywords can be input into the second language model. The semantic understanding capabilities of the second language model will then be used to identify whether each candidate sample keyword can be used as a keyword in the sample text. It is understandable that the second language model can handle various types of text processing tasks. When inputting the sample text and candidate sample keywords into the second language model, corresponding preset prompts are needed to indicate that the text processing task to be performed by the second language model is to identify whether each candidate sample keyword is a keyword in the sample text. For example, the preset prompts could be: "Please analyze whether the following words are keywords in the input text, words are: <candidate sample keyword 1>, <candidate sample keyword 2>...; text is: <sample text>". In this case, the sample text is "A cat is running", and the candidate sample keywords include four candidate sample keywords: "a", "cat", "in", and "running". After filling the sample text and candidate sample keywords into the corresponding positions in the preset prompts, they can be input into the second language model. Of course, in one possible embodiment, the preset prompt words may also include rule description keywords, which are used to instruct the second language model how to determine whether a candidate sample keyword is a sample keyword. This allows the second language model to further determine whether to identify a candidate sample keyword as a sample keyword according to corresponding rules. Then, based on the output of the second language model, the sample keywords corresponding to the sample text are determined. For example, in the example above, the output of the second language model indicates that "cat" and "running" are keywords in the sample text; therefore, "cat" and "running" are taken as the corresponding sample keywords.

[0073] Understandably, at this point, the second language model only needs to analyze the importance of each candidate keyword obtained from word segmentation in the sample text to determine the sample keywords corresponding to the sample text. This allows the second language model to focus more on the candidate keywords being processed, reducing noise interference caused by the context of the candidate keywords, and reducing interference from sentence structure, grammar, and other factors in the sample text on keyword recognition. In addition, it can avoid missing the recognition of a word in the sample text, thus reducing the difficulty of keyword recognition and improving the accuracy of the obtained sample keywords. Based on this, in this embodiment, the sample keywords can be used as the real keywords corresponding to the sample text.

[0074] After identifying the sample keywords corresponding to the sample text from the candidate sample keywords using the second language model, a training set for training the first language model can be constructed based on the sample text and its corresponding sample keywords. Specifically, each sample text and its corresponding sample keyword can be represented in the following form: [sample text, sample keyword 1, sample keyword 2, ...]. Then, the first language model is fine-tuned using the training set constructed based on the sample text and sample keywords. Specifically, the sample text is filled into a preset prompt template as input to the first language model, enabling the first language model to extract keywords from the sample text based on the analysis dimensions indicated by the preset prompt template. The output of the first language model is then determined as the predicted sample keywords. Next, the difference between the predicted sample keywords output by the first language model and the actual sample keywords is used to calculate the corresponding loss function. Backpropagation is then performed based on the loss function to calculate the gradients corresponding to each parameter of the first language model, thereby optimizing the parameters of the first language model. Specifically, the loss function can be calculated using parameters such as cosine similarity and Euclidean distance between the feature vectors corresponding to the predicted sample keywords output by the first language model and the feature vectors of the sample keywords. This is not limited in this embodiment. In this way, the first language model can learn how to accurately extract the corresponding keywords directly from the input first image description text, using them as keyword tags corresponding to the first image.

[0075] Step 203: Determine the target category label corresponding to the first image using a multi-label classification model.

[0076] A multi-label classification model is a neural network model used to determine the category of a target image to which the first image belongs. Referring to Figure 3, a multi-label classification model can be composed of multiple label sub-models. Each label sub-model can correspond to a candidate category label, and different candidate category labels correspond to different label sub-models. Each label sub-model can be a binary classifier used to determine the matching degree between its corresponding candidate category label and the first image.

[0077] The target category label is used to characterize the image category to which the first image belongs. It can be understood that a single first image can contain multiple image elements at the same time, and these image elements can correspond to multiple attributes of different dimensions such as hair and facial expressions. Based on this, the target category label can be one or more of a number of preset candidate category labels.

[0078] Candidate category labels can be preset image categories based on image categories that frequently appear during image retrieval. For example, in actual image retrieval, images in categories such as avatars and pets are frequently retrieved. Considering the broad scope of these categories, in this embodiment, these frequently retrieved image categories can be further subdivided into finer-grained categories to obtain preset candidate category labels. For instance, for the avatar category, it can be further subdivided into more fine-grained category labels such as anime avatars, human avatars, long-haired avatars, and black-haired avatars as preset candidate category labels. Similarly, for the pet category, it can be further subdivided into more fine-grained category labels such as cartoon pets, pet cats, and pet dogs as preset candidate category labels. It is understood that multiple candidate category labels correspond to categories when classifying images according to different dimensions, and the classification results of images classified according to different dimensions can overlap. That is, the images corresponding to multiple preset candidate category labels can overlap; for example, multiple candidate category labels such as anime avatars, black-haired avatars, and long-haired avatars can simultaneously correspond to the same image. Multiple pre-defined candidate category labels can form a category label set, which is stored in the database where the multi-label classification model is deployed.

[0079] It is easy to understand that in step 202, the keyword tags are obtained by extracting keywords from the first image description text. Keyword tags can only be words contained in the first image description text. However, the same image element can often be described by multiple words with similar semantics, and the first image description text may omit descriptions of certain image elements in the first image. In this case, if only keyword tags are used as tags associated with the first image, then when other users search for images, using synonyms or higher-level words of the keyword tags of a certain first image as image description text, they will be unable to retrieve that first image, leading to a decrease in the recall and precision of image retrieval. For example, an image with apples as its main image content may have keyword tags including "apple" but not "fruit." When a user uses "fruit" as the image description text for image retrieval, they will be unable to retrieve that image. Based on this, in this embodiment, frequently occurring search terms during image retrieval are also preset as candidate category tags. Through a multi-tag classification model, candidate category tags with a high semantic matching degree with the first image are determined from the preset multiple candidate category tags as the target category tags corresponding to the first image.

[0080] Specifically, in one embodiment, referring to Figure 3, after obtaining the first image, the first image and the candidate category labels can be encoded into corresponding feature vectors respectively. Then, the first image and each candidate category label are fused to obtain a feature vector corresponding to the first image and a single candidate category label. Specifically, the feature vector corresponding to the first image and the feature vector corresponding to the candidate category label can be fused by feature vector concatenation, weighted summation, or cross-attention mechanisms. Then, the result of feature fusion is used as input to the label sub-model corresponding to the candidate category label. The label sub-model calculates a probability value in the range of 0 to 1 based on the input feature fusion result. This probability value can be used as the matching degree between the first image and the corresponding candidate category label.

[0081] After determining the matching degree between the first image and each candidate category, the target category label corresponding to the first image can be determined from multiple candidate category labels based on certain rules according to the matching degree. For example, a matching degree threshold can be preset, and the candidate category labels with matching degrees greater than the threshold can be determined as the target category labels of the first image. Alternatively, in one embodiment, multiple candidate category labels can be sorted in descending order of their matching degrees, and the top K candidate category labels can be determined as the target category labels based on the sorting results, where K is a preset constant. Or, in another embodiment, multiple candidate category labels can be sorted in descending order of their matching degrees, and combined with the preset matching degree threshold, multiple candidate category labels with matching degrees greater than the preset matching degree threshold among the top K candidate category labels can be determined as the target category labels corresponding to the first image, and the target category labels can be determined as the associated labels corresponding to the first image. This improves the richness of the labels corresponding to the first image while ensuring a high semantic similarity between the associated labels and the first image, ensuring that the associated labels accurately reflect the content of the first image.

[0082] In one possible implementation, step 203, determining the target category label corresponding to the first image through a multi-label classification model, includes: forming image label pairs by combining the first image and each candidate category label, wherein a single image label pair includes the first image and a single candidate category label; for each image label pair, inputting the image label pair into a label sub-model corresponding to the candidate category label in the image label pair, so as to determine the matching degree between the first image and the candidate category label in the image label pair through the label sub-model; and determining the target category label corresponding to the first image from multiple candidate category labels based on the matching degree between the first image and each candidate category label.

[0083] Based on this, by constructing corresponding image label pairs from the first image and each candidate category label, and then inputting each image label pair into the corresponding label sub-model, the matching degree between the first image and the candidate category label is determined by the label sub-model. This decomposes the complex multi-label classification task into a binary classification task for each candidate category label, reducing the difficulty of determining the target category label of the first image and improving the accuracy of the determined target category label.

[0084] Specifically, referring to the relevant descriptions in the above embodiments, after presetting multiple candidate category labels, a category label set is constructed based on the preset candidate category labels. At this time, the category label set can be traversed, and during the traversal, each candidate category label encountered is combined with the first image to form an image label pair. For example, in one embodiment, 375 candidate category labels are preset based on dimensions such as wallpaper, avatar, and pet, such as anime character avatar, long-haired avatar, and pet dog. At this time, the category label set composed of these 375 candidate category labels is traversed, and when each candidate category label is encountered, the candidate category label is combined with the first image to form an image label pair. After traversing the category label set, 375 image label pairs are obtained, and each image label pair consists of the first image and a single candidate category label.

[0085] After obtaining multiple image label pairs, each image label pair can be input into a label sub-model corresponding to the candidate category label in that image label pair. Specifically, each label sub-model can be a binary classifier implemented based on either logistic regression or support vector machine algorithms. After training, each label sub-model can be used to calculate the matching degree between the first image and the corresponding candidate category label. The training process of the label sub-model will be explained later and will not be elaborated here. After obtaining the matching degrees output by each label sub-model, the K candidate category labels with the highest matching degrees can be determined as the target category labels corresponding to the first image.

[0086] In one possible implementation, the training process of the multi-label classification model includes: acquiring a first training sample set, wherein the first training sample set includes multiple first sample images and sample image labels corresponding to each first sample image, and the sample image labels include at least one of multiple candidate category labels; forming sample image label pairs by combining the first sample images with each candidate category label, wherein a single sample image label pair includes a first sample image and a single candidate category label; for each sample image label pair, inputting the sample image label pair into the label sub-model corresponding to the candidate category label in the sample image label pair, so as to determine the sample matching degree between the first sample image and the candidate category label in the sample image label pair through the label sub-model; determining a first loss value corresponding to the label sub-model based on the sample matching degree and the sample image label corresponding to the first sample image, and optimizing the parameters of the label sub-model based on the first loss value.

[0087] Based on this, by inputting the first sample image into each label sub-model, each label sub-model calculates the matching degree between the first sample image and the candidate category label corresponding to the label sub-model. In this way, the multi-label classification model predicts multiple predicted sample category labels corresponding to the first sample image. Then, the loss function is calculated by the difference between the predicted sample category label and the real sample label corresponding to the first sample image to obtain the first loss value corresponding to each label sub-model. The parameters of each label sub-model are then optimized so that each label sub-model can learn how to calculate the matching degree between the image and the corresponding candidate category label.

[0088] In this embodiment, the first sample image is a training sample used to train the multi-label classification model. The first sample image has been labeled, and each first sample image is assigned a corresponding sample image label. These sample image labels are one or more of multiple candidate category labels. Each first sample image only matches with several candidate category labels that serve as its sample image label, and does not match with other candidate category labels.

[0089] After acquiring the first sample image, for each candidate category label, each first sample image and the candidate category label are combined into a sample image label pair, which is then used as the input to the label sub-model corresponding to the candidate category label. Specifically, the first sample image and the candidate category label are encoded into corresponding feature vectors, and feature fusion processing is performed. The vector obtained by feature fusion is input into the corresponding label sub-model so that the sample matching degree between the candidate category label and each first sample image can be calculated through the corresponding label sub-model.

[0090] After obtaining the sample matching degree between the corresponding candidate category label and each first sample image for each label sub-model, the true matching degree between each first sample image and each candidate category label can be determined based on the sample image label corresponding to each first sample image. That is, the true matching degree between the candidate category label corresponding to the label sub-model and each first sample image can be obtained. Then, based on the difference between the true matching degree and the sample matching degree calculated by the label sub-model, the corresponding first loss value can be calculated, and backpropagation can be performed based on the first loss value to optimize the parameters of the label sub-model.

[0091] Furthermore, in this embodiment, all candidate category labels are trained using the same training samples. This avoids a situation where a certain label sub-model cannot accurately calculate the matching degree between the corresponding candidate category label and the image due to having fewer training samples than other label sub-models, which would lead to a decrease in the accuracy of the multi-label classification model in determining the category label of the input image. This ensures the accuracy of the category labels determined by the trained multi-label classification model.

[0092] In one possible implementation, the first loss value corresponding to the label sub-model is determined based on the sample matching degree and the sample image label corresponding to the first sample image, including: for each candidate category label, determining the true matching degree between the candidate category label and the first sample image based on the sample image label corresponding to the first sample image; for each label sub-model, determining the first loss value corresponding to the label sub-model based on the true matching degree between its corresponding candidate category label and each first sample image, and the sample matching degree between its corresponding candidate category label and each first sample image.

[0093] Based on this, the true matching degree between each first sample image and each candidate category label is determined according to the sample image label corresponding to each first sample image; then, based on the difference between the sample matching degree between the corresponding candidate category label predicted by each label sub-model and each first sample label and the corresponding true matching degree, the first loss value used when training the label sub-model is determined. In this way, the label sub-model is guided to accurately predict the matching degree between the input image and its corresponding candidate category label, thereby improving the label sub-model's ability to predict the matching degree between its corresponding candidate category label and the image.

[0094] Specifically, the sample image label corresponding to the first sample image indicates which candidate category labels the first sample image actually matches and which do not. This determines the true matching degree between the first sample image and each candidate category label. Specifically, the true matching degree between the first sample image and its corresponding sample image labels can be set to 1, while the true matching degree with other candidate category labels can be set to 0. Then, for each label sub-model, the corresponding first loss value can be calculated based on the sample matching degree and the true matching degree using the following formula:

[0095] Where L is the first loss value to be calculated, and N is the number of the first sample images; y i p represents the true match between the i-th first sample image and the candidate category label corresponding to the label sub-model, and its value is 0 or 1; iIt is the sample matching degree between the i-th first sample image output by the label sub-model and its corresponding candidate category label.

[0096] In one possible implementation, optimizing the parameters of the labeled sub-model based on the first loss value includes: determining the update gradient corresponding to each model parameter of the labeled sub-model based on the first loss value; for each model parameter in the labeled sub-model, determining the update step size corresponding to the model parameter based on the update gradient and historical gradient of the model parameter, and updating the model parameter based on the update step size, wherein the historical gradient is the gradient of the model parameter in the previous iteration.

[0097] Based on this, by combining the historical gradients of the labeled sub-model during previous parameter iterations, the update step size for the current parameter iteration is calculated, and the parameters of the labeled sub-model are updated based on the update step size. Thus, by combining the gradient of the current iteration and the gradients of the historical iterations to control the update step size of the labeled sub-model, the convergence speed of the labeled sub-model can be accelerated and the training efficiency can be improved.

[0098] Specifically, backpropagation is performed based on the first loss value, and the partial derivatives of the first loss value with respect to each parameter in the labeled sub-model are calculated to obtain the update gradients corresponding to each model parameter. Before the current training round, the labeled sub-model may have undergone multiple training rounds and performed multiple gradient descent processes. During each training round, the descent gradient of the labeled sub-model in the current round can be recorded, and a gradient list corresponding to the labeled sub-model can be constructed. The historical gradients of the labeled sub-model can then be obtained from this gradient list. Specifically, this can be similar to the method used in the Adam optimizer to control the update step size of the model parameters based on historical gradients and the descent gradient of the current round, which will not be elaborated upon here.

[0099] In one possible implementation, the method further includes: in response to a newly added candidate category label, obtaining an original label sub-model corresponding to the newly added candidate category label; obtaining a second training sample set, wherein the second training sample set includes multiple second sample images, and the second sample images are set with a true matching degree between them and the newly added candidate category label; determining the predicted matching degree between each second sample image and the newly added candidate category label using the original label sub-model, and optimizing the parameters of the original label sub-model based on the predicted matching degree and the true matching degree between each second sample image and the newly added candidate category label, until a trained original label sub-model is obtained; adding the trained original label sub-model to a multi-label classification model, and determining the newly added candidate category label as the candidate category label.

[0100] Based on this, when adding a new candidate category label, it is only necessary to train a label sub-model corresponding to the new candidate category label and add the trained new label sub-model to the multi-label classification model, without having to retrain the label sub-models corresponding to other candidate category labels, thereby reducing the complexity of adding new candidate category labels.

[0101] It is understood that in this embodiment, the candidate category labels are defined based on the high-frequency words used when performing image retrieval. They can only cover most of the images that users will frequently search. Therefore, it is often necessary to add candidate category labels based on users' search behavior to provide more accurate image classification and labeling services.

[0102] Therefore, in this embodiment, when a new candidate category label needs to be added, since each label sub-model in the multi-label classification model is dedicated to determining the matching degree between the image and an existing candidate category label, when a new candidate category label needs to be added, it is necessary to obtain the original label sub-model corresponding to the newly added candidate category label. Specifically, a new binary classifier is generated as the original label sub-model corresponding to the newly added candidate category label. The original label sub-model is the label sub-model initialized before training.

[0103] To enable the original label sub-model to accurately calculate the matching degree between the image and the newly added candidate category label, the original label sub-model needs to be trained. Specifically, a second training sample set is first obtained, which includes multiple second sample images, and each second sample image is labeled with whether it matches the newly added candidate category label.

[0104] In one embodiment, the second training sample set may be obtained by annotating the images in the first training sample set, that is, the second training sample set may be composed of the images in the first training sample set.

[0105] After obtaining the second training sample set, the second sample images and the newly added candidate category labels can be encoded into corresponding feature vectors. Then, the feature vectors corresponding to the second sample images and the newly added candidate category labels are fused through concatenation, weighted summation, etc. The feature fusion result is input into the original label sub-model to calculate the predicted matching degree between the second sample images and the newly added candidate category labels. Since each second sample image is labeled with its true matching degree with the newly added candidate category label, the corresponding loss value can be calculated based on the difference between the predicted matching degree and the true matching degree. Specifically, the method for calculating the loss value can refer to the method for calculating the first loss value in the above steps, which will not be repeated here. After calculating the corresponding loss value, backpropagation is performed based on the loss value to optimize the parameters of the original label sub-model until the preset training completion conditions are met, and the trained original label sub-model is obtained.

[0106] After obtaining the trained original label sub-models, referring to Figure 5, since the label sub-models corresponding to each candidate category label process data independently, the trained original label sub-models are directly added to the multi-label classification model as a processing branch, and the newly added candidate category labels are used as candidate category labels. For example, referring to Figure 5, the multi-label classification model originally includes N label sub-models, each corresponding to one of the N candidate category labels. After the original label sub-models corresponding to the newly added candidate category labels are trained, they are added to the multi-label classification model as label sub-model N+1, and the newly added candidate category labels are used as candidate category labels N+1.

[0107] In one possible implementation, the image label pair is input into a label sub-model corresponding to the candidate category label in the image label pair, so as to determine the matching degree between the first image and the candidate category label in the image label pair through the label sub-model. This includes: encoding the candidate category label in the image label pair into candidate category label features through a first text encoder, and encoding the first image into first image features through a first image encoder, wherein the first text encoder and the first image encoder are jointly trained; concatenating the candidate category label features and the first image features to obtain concatenated features, and inputting the concatenated features into the label sub-model corresponding to the candidate category label in the image label pair, so as to determine the matching degree between the first image and the candidate category label in the image label pair through the label sub-model.

[0108] Based on this, the first image and the candidate category label are encoded by the first image encoder and the first text encoder that have been jointly trained, respectively, to obtain the first image features and the candidate category label features aligned in the embedding space. In this way, by concatenating the first image features and the candidate category label features and inputting the concatenation result into the corresponding label sub-model, the matching degree between the first image and the corresponding candidate category label can be accurately calculated by the corresponding label sub-model, so as to determine the target category label corresponding to the first image from multiple candidate category labels based on the matching degree.

[0109] The first text encoder is an encoder used to encode features of text modal input. Specifically, it can be a text modal encoder based on a convolutional neural network, or it can be a text encoder based on a language model such as Bidirectional Encoder Representations from Transformers (BERT). This embodiment does not limit the specific type of encoder. The first image encoder can be an encoder based on a convolutional neural network, for example, an encoder based on a Residual Network (RseNet).

[0110] By using a first image encoder and a first text encoder to encode features of the first image and candidate category labels respectively, the corresponding first image features and candidate category label features can be obtained. Then, the first image features and candidate category label features are concatenated to obtain concatenated features. These concatenated features are then input into the label sub-model corresponding to the candidate category label to calculate the corresponding matching degree. For example, in a binary classifier implemented based on a logistic regression algorithm, the model parameters of this binary classifier include a trained weight matrix. After the feature vector is input into the binary classifier, linear regression processing is performed on the input feature vector based on this weight matrix. Then, the sigmoid function is used to map the result of the linear regression processing to the range (0, 1), thereby obtaining the corresponding matching degree.

[0111] In one embodiment, both the candidate category label feature and the first image feature are 1024-dimensional feature vectors. Thus, the high-dimensional feature space allows both the first image feature and the candidate category label feature to provide rich information, enabling the corresponding label sub-model to capture the similarity between the first image and the candidate category label based on relatively rich information.

[0112] In one possible implementation, the training process of the first text encoder and the first image encoder includes: acquiring a third training sample set, which includes multiple sample image-text pairs, each sample image-text pair including a third sample image and sample descriptive text corresponding to the third sample image; using the first text encoder and the first image encoder to perform feature encoding on each sample descriptive text and each third sample image, respectively, to obtain multiple sample descriptive text features and multiple sample image features; determining the predicted matching degree between each sample descriptive text feature and each sample image feature, and based on the sample image-text pairs, determining the true matching degree between the sample descriptive text features and each sample image feature; determining a second loss value based on the difference between the predicted matching degree and the true matching degree, and optimizing the parameters of the first text encoder and the first image encoder based on the second loss value.

[0113] Based on this, by using a third sample image and the corresponding sample description text, the first text encoder and the first image encoder are jointly trained, enabling the first text encoder and the first image encoder to learn how to encode text data and image data with similar semantics into text feature vectors and image feature vectors with high matching degree, respectively. In this way, when the first text encoder and the first image encoder encode the candidate category labels and the first image respectively, text features and image features that are aligned in the embedding space and can accurately represent the matching degree between the candidate category labels and the first image can be obtained, so as to determine the matching score between the first image and each candidate category label.

[0114] The third training sample set is a dataset consisting of multiple sample image-text pairs. Each sample image-text pair includes a corresponding third sample image and sample description text. The sample description text is used to describe the content in the third sample image. For example, if the third sample image is a person image, the corresponding sample description text can be used to describe the person's appearance, expression, and other features.

[0115] After obtaining the third training sample set, each image-text pair can be sorted to determine its corresponding index. Then, each third sample image is input into the first image encoder for feature encoding, resulting in multiple sample image features. Similarly, each sample descriptive text is input into the first text encoder for feature encoding, resulting in multiple sample descriptive text features. Next, for each third sample image, the cosine similarity or dot product between the corresponding sample image features and each sample descriptive text feature is calculated to obtain the similarity between the third sample image and each sample descriptive text. Softmax processing is then applied to the similarity to obtain the predicted matching degree between the third sample image and each sample descriptive text. Likewise, for each sample descriptive text, the similarity between the corresponding sample descriptive text features and each sample image features is calculated and softmax processing is applied to obtain the predicted matching degree between the sample descriptive text and each third sample image. These predicted matching degrees can be represented as vectors for subsequent loss function calculation.

[0116] Based on the matching relationship indicated by the sample image-text pairs, the true matching probability between the third sample image and the sample description text is determined. It can be understood that since the sample description text in the i-th sample image-text pair only matches the third sample image in the i-th sample image-text pair, but does not match the third sample images in other sample image-text pairs, then in the vector used to characterize the true matching degree of the i-th sample description text, only the i-th element has a value of 1, and the values ​​of the remaining elements are all 0. For example, the true matching degree of the first sample description text can be represented as [1,0,0,...0]. Similarly, the true matching degree between each third sample image and each sample description text can be obtained.

[0117] Then, a second loss value can be calculated based on the difference between the predicted and actual matching degrees between the sample description text and the third sample image. Specifically, referring to Figure 6, the second loss value can be calculated based on the cross-entropy between the predicted and actual matching degrees. The cross-entropy function is as follows:

[0118] in, It represents the true matching degree between the text features of the i-th sample and the image features of each sample. It represents the predicted matching degree between the text features of the i-th sample and the image features of each sample. It represents the true matching degree between the image features of the j-th sample and the descriptive text features of each sample. It represents the predicted matching degree between the image features of the j-th sample and the descriptive text features of each sample. It represents the true matching degree between the text features of the i-th sample and the image features of the x-th sample; It is the predicted matching degree between the text features of the i-th sample and the image features of the x-th sample; It is the true matching degree between the image features of the j-th sample and the descriptive text features of the x-th sample; is the predicted matching degree between the features of the j-th sample image and the features of the x-th sample description text; log represents the logarithmic function; B is the number of third sample images, i.e., the number of sample description texts. Since each third sample image actually has a true matching degree of 1 with only one sample description text and a true matching degree of 0 with other sample description texts, and each sample description text actually has a true matching degree of 1 with only one third sample image and a true matching degree of 0 with other third sample images, we can conclude that:

[0119] Due to the predicted matching degree and The elements in the expression are all obtained after normalization, and their values ​​are in the range [0,1]. From the above formula, it is easy to see that when... The larger the cross-entropy function, the higher the similarity between the sample image features and the sample description text features generated by the third sample image and sample description text in the same sample image-text pair. In this case, the value of the cross-entropy function is smaller, and the value of the second loss is also smaller. Based on this, by backpropagating based on the second loss value to minimize the second loss value, the first image encoder and the first text encoder can learn how to encode text and images with high semantic similarity into feature vectors with high semantic similarity.

[0120] Step 204: Determine the keyword tags and target category tags as the associated tags for the first image.

[0121] Specifically, after obtaining the keyword tags and target category tags corresponding to the first image, these tags can reflect the content of the first image to a certain extent and have a direct semantic connection with it. Therefore, these keyword tags and target category tags can be determined as the associated tags corresponding to the first image, so that the first image can be retrieved based on the associated tags when image retrieval is needed later. Specifically, after determining the associated tags corresponding to the first image, these associated tags and the first image can be associated and stored in a database. For example, a tag table and an image table can be created in a relational database. After converting the first image and its corresponding associated tags into a form suitable for storage in a relational database, the first image and its corresponding associated tags are stored in the image table and the tag table respectively, and an index relationship between the first image and the associated tags is established using foreign keys. Then, the first image can be retrieved from the database based on the associated tags corresponding to it.

[0122] In the embodiments disclosed in steps 201 to 204, keyword extraction is performed on the first image description text used to describe the first image to obtain keyword tags for the first image, ensuring that there is a direct semantic connection between the keyword tags and the first image. At the same time, the semantic matching degree between the first image and each candidate category tag is calculated through a multi-label classification model, and multiple candidate category tags with high semantic matching degrees are determined as the target category tags corresponding to the first image, ensuring that the target category tags and the first image are semantically related. Then, the keyword tags and target category tags are determined as the associated tags corresponding to the first image. In this way, while ensuring the richness of the associated tags of the first image, it can be ensured that these associated tags are also semantically related to the first image.

[0123] Furthermore, in the image tag generation method of this application, keywords are extracted from the description text of the first image, and the extracted keywords are used as the associated tags of the first image, thereby ensuring that the generated associated tags can better reflect the characteristics of the first image itself; at the same time, a target category tag corresponding to the first image is determined from multiple preset candidate category tags through a multi-tag classification model, and used as an additional associated tag for the first image, ensuring that the additional associated tags are standardized, thereby classifying the first image in a standardized manner.

[0124] In one possible implementation, after determining the keyword tags and target category tags as associated tags for the first image, the method includes: receiving target image description text in response to an image retrieval request; determining target keyword tags based on the target image description text, and determining target associated tags based on the target keyword tags, wherein the target associated tags are at least one of the associated tags corresponding to each first image, and the semantic similarity between the target keyword tags and the target associated tags satisfies a preset semantic similarity condition; retrieving a target image from each first image based on the target keyword tags and the target associated tags, and displaying the target image, wherein the associated tags corresponding to the target image include at least a plurality of the target keyword tags and the target associated tags.

[0125] Based on this, upon receiving an image retrieval request from a user, the system receives the target image description text entered by the user on the image retrieval page and extracts target keyword tags to characterize the features of the image the user wishes to retrieve. Based on the semantic similarity between the target keyword tags and the associated tags corresponding to each first image, the associated tags with higher semantic similarity are identified as target associated tags. Image retrieval is then performed based on the target keyword tags and target associated tags to further enrich the image retrieval results, while ensuring that the retrieved images semantically match the entered target image description text.

[0126] An image retrieval request is a request initiated by a user to the server to retrieve an image associated with the text entered by the user. An image retrieval page is a page that provides image retrieval services to the user; it can be an application interface or a web page. The target image description text is text entered by the user to describe the content of the image they wish to retrieve. It is understood that an image retrieval page should at least include a text input box for the user to enter the text used for image retrieval and a button for the user to initiate an image retrieval request to the server. Referring to Figure 7, the user can initiate an image retrieval request by accessing an application for image retrieval through a user terminal.

[0127] Specifically, referring to Figure 8, after opening the application, the user can select the provided gallery application to access the image search page, enter text describing the image to be searched in the corresponding text input box, and click the button on the page to initiate an image search request to the server; then, the server reads the text entered by the user in the text input box on the image search page as the target image description text.

[0128] The target keyword tag is a keyword extracted from the target image description text. Specifically, referring to Figure 3, the target image description text can be filled into a preset prompt template and input into the first language model. The first language model performs semantic analysis on the target image description text based on the thought chain formed by the various logical analysis prompt words in the preset prompt template, so as to extract the keywords of the target image description text as target keyword tags. The details of extracting target keyword tags from the target image description text can be referred to the relevant description of step 202 above, and will not be repeated here.

[0129] After extracting the target keyword tags, considering that the target keyword tags are words extracted from the description text of the target images, if image retrieval is performed solely based on the target keyword tags, the number of retrieved images will be relatively small, and the recall rate of image retrieval will be relatively low. Therefore, in this embodiment, corresponding target association tags are also determined based on the target keyword tags. Specifically, the semantic similarity between the target keyword tags and the association tags corresponding to each first image stored in the database can be calculated, and association tags with higher semantic similarity to the target keyword tags are determined as target association tags. It is understood that, referring to the relevant descriptions in the above embodiments, when storing each first image in the database, each first image and its corresponding association tag are associated and stored in a relational database. In image retrieval, the tags used to retrieve the corresponding first image from the database are already determined. For example, if the image database stores 100,000 first images, each with 15 associated tags, after deduplication of the associated tags for each first image, there are 7,000 unique associated tags. Then, the semantic similarity between the target keyword tag and these 7,000 associated tags can be calculated. For example, the target keyword tag and each associated tag can be encoded into corresponding feature vectors, and then cosine similarity or Euclidean distance can be calculated based on the feature vectors corresponding to the target keyword tag and each associated tag to determine the semantic similarity between the target keyword tag and each associated tag. After that, the tags are sorted according to semantic similarity, and the top N associated tags with the highest semantic similarity to the target keyword tag are determined as target associated tags for subsequent image retrieval based on the target keyword tag and target associated tags.

[0130] In another embodiment, considering that the computational cost required to determine the target associated label by calculating the semantic similarity between the target keyword label and each associated label may be relatively large, the target associated label can also be determined from each associated label corresponding to each first image by means of an approximate nearest neighbor search method.

[0131] After determining the target keyword tags and target association tags, the system can retrieve the first image corresponding to the target association tags and target keyword tags as the target image, and return the retrieved target images to the user's page for display.

[0132] Understandably, for any target image, a high degree of overlap between the associated tags, target keyword tags, and associated tags indicates that the target image better matches the content described in the image description text compared to other target images, thus better meeting the user's search needs. Based on this, when displaying target images, the retrieved images can be sorted according to the overlap between the associated tags, target keyword tags, and associated tags, and the images are displayed based on the sorted results. This prioritizes images that match multiple target keyword tags and associated tags, making the display results more user-friendly. For example, if the number of target keyword tags and associated tags obtained from the image description text is 10, and among the retrieved target images, image A's associated tags contain 7 of these 10 tags, while another target image B's associated tags only contain 3, then image A will be prioritized for display on the user's interface.

[0133] In one possible implementation, determining target keyword tags based on the target image description text includes: obtaining the object identifier corresponding to the image retrieval request, where the object identifier is the identity identifier corresponding to the object that initiated the image retrieval request; determining at least one historical retrieval text based on the object identifier; and inputting the historical retrieval text, the target image description text, and the preset prompt template into a first language model to perform semantic analysis on the historical retrieval text and the target image description text through the first language model to obtain the target keyword tags.

[0134] Based on this, when performing image retrieval, the user's historical search text is combined to determine the user's preferences when performing image retrieval, and the image retrieval is performed accordingly, so that the image retrieval results are more consistent with the user's historical search behavior, thereby further improving the accuracy of image retrieval.

[0135] The object identifier can be the account identity identifier corresponding to the user's logged-in account when initiating an image search request on the image search page; alternatively, it can be the IP address identifier when the user accesses the image search page as a guest. Specifically, when a user initiates an image search request to the server, the request itself often carries a corresponding object identifier so that the server can identify which user initiated the request and determine which user's interface the image search results should be returned to. Specifically, a specific field can be set in the image search request to populate the object identifier. When the server receives the image search request, it can retrieve the corresponding object identifier by searching the specific field within the request.

[0136] Historical search text refers to the search text carried in previous image search requests initiated by the user corresponding to the object identifier. It can be understood that this historical search text can reflect the user's preferences when conducting image searches, especially when the target keyword tag is a polysemous word. Based on the historical search text, the actual meaning represented by the target keyword tag can be determined. For example, a target keyword tag might correspond to both the name of a crop and the abbreviation of a digital technology company. If the user's historical search text indicates that the user frequently searches for images related to digital technology, then the target keyword tag extracted from the target image search text can be determined to actually refer to the meaning of the word as a digital technology company. Subsequent image searches can then be conducted based on this, avoiding images corresponding to the target keyword tag as a crop name as the searched images, thereby improving the accuracy of image retrieval.

[0137] Specifically, the preset prompt template can also include prompt words for filling in historical search text. For example, the preset prompt template can include "This user's historical search records include: <historical search text>. Please analyze the following text based on the user's historical search records: <target image description text>". Based on this, the first language model can combine the user's historical search text to perform semantic analysis on the target image search text, while avoiding confusion between historical search text and target image description text, which would lead to the extraction of words in the historical search text as target keyword tags, thereby improving the accuracy of the extracted target keyword tags and thus improving the accuracy of image retrieval.

[0138] In one possible implementation, displaying a target image includes: clustering the target images based on target association tags and target keyword tags, wherein a single target image corresponds to at least one of the target association tags and target keyword tags; displaying the target image, target association tags, and target keyword tags; and, in response to the selection of a target association tag or target keyword tag, displaying the target image corresponding to the selected target association tag or target keyword tag.

[0139] Based on this, when displaying target images, the retrieved target images are clustered based on the target association tags or target keyword tags corresponding to each target image, and the target images, target association tags, and target keyword tags are displayed simultaneously on the interface. When the user selects a target association tag or target keyword tag, the display results are adjusted to only display the target images corresponding to the selected tags, so that the displayed target images better meet the user's search needs.

[0140] Specifically, after image retrieval based on target keyword tags and target association tags, the obtained target images can correspond to at least one of the target keyword tags and target association tags. Based on this, the target images can be clustered according to the association tags corresponding to each target image. That is, multiple target images corresponding to the same target keyword tag or target association tag are considered as a cluster result, thus obtaining the clustering results of target images corresponding to each target keyword tag or target association tag. For example, if 300 target images in the retrieved target images correspond to target keyword tag 1, then these 300 target images are clustered into the clustering result corresponding to target keyword tag 1.

[0141] It should be noted that in this embodiment, a single target image can simultaneously correspond to multiple target keyword tags and target association tags. Based on this, a single target image can belong to multiple clustering results simultaneously. For example, if a retrieved target image simultaneously corresponds to target keyword tag 1, target keyword tag 3, and target association tag 1, then the target image simultaneously belongs to the clustering results corresponding to target keyword tag 1, target keyword tag 3, and target association tag 1.

[0142] Once the target image is retrieved, the retrieved target images, target keyword tags, and target association tags can be displayed on the user terminal interface. Specifically, the user terminal interface can be set with a tag display area and an image display area. The target keyword tags and target association tags are displayed in the tag display area, while the target images are displayed in the image display area according to certain rules.

[0143] After displaying the target image, various target keyword tags, and various target association tags, the user may perform a secondary search to further filter images from the retrieved target images to match the user's search intent. At this time, the user can select target keyword tags and target association tags displayed on the user's terminal page by clicking on them. The server can respond to the user's selection of target keyword tags or target association tags, and based on the clustering results of the target images, display the target image corresponding to the selected target keyword tag or target association tag on the user's terminal page, thus making the displayed target image more consistent with the user's search intent. For example, referring to Figure 9, after a user searches for an avatar, multiple tags such as "young people," "anime," "white shirt," and "short hair" are displayed in the tag display area of ​​the interface. After the user clicks on the corresponding tag, the target image corresponding to the selected tag will be displayed on the interface. For example, referring to Figure 9, when the user selects the tag "short hair" or "anime," the target image corresponding to the "short hair" or "anime" tag from the target images retrieved based on the avatar will be displayed on the interface.

[0144] For example, the image tag generation method provided in this disclosure can be used for image search. Specifically, in image library software, it is often necessary to upload a large number of images to the software's database in advance and add corresponding associative tags to these images. At this time, the image tag generation method of this disclosure can be used to generate image tags corresponding to each image, and then each image and its corresponding image tags can be stored in the database. This allows users to retrieve corresponding images by inputting text when using image library software.

[0145] For example, the image tag generation method of this disclosure can be applied to user dynamic image matching. Specifically, when a user posts a dynamic on some Internet platforms or communities, they often add appropriate images to the dynamic to improve the viewing experience. At this time, the application of the Internet platform or community can automatically obtain the dynamic edited by the user, use the dynamic edited by the user as the target image description text, perform image retrieval, and display the retrieved target image on the user's terminal interface so that the user can select the corresponding image as the image matching for the user dynamic.

[0146] For example, the image tag generation method of this disclosure can be applied to emoticon recommendation. Specifically, in online conversations, using emoticons can often better express the user's meaning and enhance the interaction between users. At this time, the conversation software can use the emoticon as the first image, generate the associated tags corresponding to each emoticon through the image tag generation method of this disclosure, and store them in the database of the conversation software. When the conversation text entered by the user in the conversation text input box is obtained, it is used as the target image retrieval text to perform image retrieval, thereby displaying the corresponding emoticons in the interface of the conversation software for the user to choose from.

[0147] Referring to Figure 10, Figure 10 illustrates an image tag generation apparatus 1000 provided in an embodiment of this disclosure. The apparatus 1000 includes:

[0148] The acquisition unit 1010 is used to acquire a first image and a first image description text for describing the first image;

[0149] The keyword extraction unit 1020 is used to extract keywords from the description text of the first image and use them as keyword tags corresponding to the first image.

[0150] The category label generation unit 1030 is used to determine the target category label corresponding to the first image through a multi-label classification model. The target category label is used to characterize the image category of the first image. The target category label includes at least one of a plurality of preset candidate category labels. The multi-label classification model includes a plurality of label sub-models corresponding to the plurality of candidate category labels respectively. Each label sub-model is used to determine the matching degree between the corresponding candidate category label and the first image. The target category label is determined based on the matching degree between each candidate category label and the first image.

[0151] The determining unit 1040 is used to determine the keyword tag and the target category tag as the associated tag of the first image.

[0152] Optionally, the keyword extraction unit 1020 is used for:

[0153] The first image description text is filled into the preset prompt template to obtain the first input text. The preset prompt template includes at least one logical analysis prompt word, which is used to indicate the analysis dimension when the first language model performs semantic analysis on the first image description text.

[0154] The first input text is fed into the first large language model, so that the first large language model performs semantic analysis on the first image description text based on the analysis dimension indicated by the logical analysis prompt words, and obtains keywords.

[0155] Optionally, the image label generation device 1000 further includes a first training unit, which is used for:

[0156] Obtain the sample text and decompose it into at least one candidate sample keyword;

[0157] The sample text and candidate sample keywords are input into the second language model to determine the sample keywords corresponding to the sample text from the candidate sample keywords.

[0158] Input the sample text and the preset prompt template into the first language model to obtain the predicted sample keywords output by the first language model, and optimize the parameters of the first language model based on the difference between the predicted sample keywords and the sample keywords.

[0159] Optionally, the category label generation unit 1030 is used for:

[0160] The first image and each candidate category label are respectively composed of image label pairs, wherein a single image label pair includes the first image and a single candidate category label;

[0161] For each image label pair, the image label pair is input into the label sub-model corresponding to the candidate category label in the image label pair, so as to determine the matching degree between the first image and the candidate category label in the image label pair through the label sub-model;

[0162] Based on the matching degree between the first image and each candidate category label, the target category label corresponding to the first image is determined from multiple candidate category labels.

[0163] Optionally, the image label generation device 1000 further includes a second training unit, which is used for:

[0164] Obtain a first training sample set, wherein the first training sample set includes a plurality of first sample images and a sample image label corresponding to each first sample image, and the sample image label includes at least one of a plurality of candidate category labels;

[0165] The first sample image is combined with each candidate category label to form a sample image label pair, wherein a single sample image label pair includes the first sample image and a single candidate category label;

[0166] For each sample image label pair, the sample image label pair is input into the label sub-model corresponding to the candidate category label in the sample image label pair, so as to determine the sample matching degree between the first sample image and the candidate category label in the sample image label pair through the label sub-model;

[0167] Based on the sample matching degree and the sample image label corresponding to the first sample image, the first loss value corresponding to the label sub-model is determined, and the parameters of the label sub-model are optimized based on the first loss value.

[0168] Optionally, the second training unit is used for:

[0169] For each candidate category label, the true matching degree between the candidate category label and the first sample image is determined based on the sample image label corresponding to the first sample image;

[0170] For each label sub-model, the first loss value corresponding to the label sub-model is determined based on the true matching degree between its corresponding candidate category label and each first sample image, and the sample matching degree between its corresponding candidate category label and each first sample image.

[0171] Optionally, the second training unit is used for:

[0172] Based on the first loss value, determine the update gradient corresponding to each model parameter of the labeled sub-model;

[0173] For each model parameter in the labeled sub-model, the update step size corresponding to the model parameter is determined based on the update gradient and historical gradient of the model parameter, and the model parameter is updated based on the update step size. The historical gradient is the gradient of the model parameter in the previous iteration.

[0174] Optionally, the image label generation apparatus 1000 further includes a new category label processing unit, which is used for:

[0175] In response to a newly added candidate category label, obtain the original label sub-model corresponding to the newly added candidate category label;

[0176] Obtain a second training sample set, which includes multiple second sample images, each of which is set with a true match degree between itself and the newly added candidate category label;

[0177] The predicted matching degree between each second sample image and the newly added candidate category label is determined through the original label sub-model. Based on the predicted matching degree and the true matching degree between each second sample image and the newly added candidate category label, the parameters of the original label sub-model are optimized until the trained original label sub-model is obtained.

[0178] The trained original label sub-model is added to the multi-label classification model, and the newly added candidate category labels are determined as candidate category labels.

[0179] Optionally, the category label generation unit 1030 is used for:

[0180] The first text encoder encodes the candidate category labels in the image label pair into candidate category label features, and the first image encoder encodes the first image into first image features. The first text encoder and the first image encoder are jointly trained.

[0181] The candidate category label features and the first image features are concatenated to obtain concatenated features. The concatenated features are then input into the label sub-model corresponding to the candidate category labels in the image label pair. The label sub-model is used to determine the matching degree between the first image and the candidate category labels in the image label pair.

[0182] Optionally, the image label generation device 1000 further includes a third training unit, which is used for:

[0183] Obtain the third training sample set, which includes multiple sample image-text pairs. Each sample image-text pair includes a third sample image and sample description text corresponding to the third sample image.

[0184] By using the first text encoder and the first image encoder, feature encoding is performed on the description text of each sample and the third sample image respectively, resulting in multiple sample description text features and multiple sample image features;

[0185] Determine the predicted matching degree between the descriptive text features and the image features of each sample, and based on the sample image-text pairs, determine the true matching degree between the descriptive text features and the image features of each sample;

[0186] A second loss value is determined based on the difference between the predicted matching degree and the actual matching degree, and the parameters of the first text encoder and the first image encoder are optimized based on the second loss value.

[0187] Optionally, the image tag generating apparatus 1000 further includes an image retrieval unit, which is used for:

[0188] In response to an image retrieval request, receive the target image description text;

[0189] Target keyword tags are determined based on the target image description text, and target association tags are determined based on the target keyword tags. The target association tags are at least one of the association tags corresponding to each first image. The semantic similarity between the target keyword tags and the target association tags satisfies the preset semantic similarity conditions.

[0190] Based on target keyword tags and target association tags, a target image is retrieved from each of the first images and the target image is displayed. The association tags corresponding to the target image include at least a number of target keyword tags and target association tags.

[0191] Optionally, the image retrieval unit is used for:

[0192] Obtain the object identifier corresponding to the image retrieval request. The object identifier is the identity identifier corresponding to the object that initiated the image retrieval request.

[0193] Based on the object identifier, at least one historical search text is identified. The historical search text, the target image description text, and the preset prompt template are then input into the first language model. The first language model performs semantic analysis on the historical search text and the target image description text to obtain the target keyword tags.

[0194] Optionally, the image retrieval unit is used for:

[0195] Clustering of target images based on target association tags and target keyword tags, wherein a single target image corresponds to at least one of the target association tags and target keyword tags;

[0196] Display the target image, target association tags, and target keyword tags;

[0197] In response to the selection of a target association tag or target keyword tag, display the target image corresponding to the selected target association tag or target keyword tag.

[0198] Referring to Figure 11, which is a partial structural block diagram of a terminal 110 implementing the image tag generation method of this embodiment, the terminal includes: a radio frequency (RF) circuit 1110, a memory 1115, an input unit 1130, a display unit 1140, a sensor 1150, an audio circuit 1160, a wireless fidelity (WiFi) module 1170, a processor 1180, and a power supply 1190, etc. Those skilled in the art will understand that the structure of the terminal 110 shown in Figure 11 does not constitute a limitation on a mobile phone or computer, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0199] The RF circuit 1110 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 1180; in addition, it transmits uplink data to the base station.

[0200] The memory 1115 can be used to store software programs and modules. The processor 1180 executes various functional applications and data processing of the content terminal by running the software programs and modules stored in the memory 1115.

[0201] The input unit 1130 can be used to receive input numeric or character information, and to generate key signal inputs related to the settings and function control of the content terminal. Specifically, the input unit 1130 may include a touch panel 1131 and other input devices 1132.

[0202] Display unit 1140 can be used to display input or provided information, as well as various menus of the content terminal. Display unit 1140 may include display panel 2941.

[0203] Audio circuitry 1160, speaker 1161, and microphone 1162 provide an audio interface.

[0204] In this embodiment, the processor 1180 included in the object terminal 110 can execute the image tag generation method of the previous embodiment.

[0205] The target terminal 110 in this disclosure includes, but is not limited to, mobile phones, computers, smart audio interaction devices, smart home appliances, vehicle terminals, and aircraft. This invention can be applied to various scenarios, including but not limited to content recommendation and data filtering.

[0206] Figure 12 is a partial structural block diagram of a server 140 implementing the image tag generation method of this disclosure. The server 140 can vary significantly due to different configurations or performance, and may include one or more central processing units (CPUs) 1222 (e.g., one or more processors) and a memory 1232, and one or more storage media 1230 (e.g., one or more mass storage devices) for storing application programs 1242 or data 1244. The memory 1232 and storage media 1230 may be temporary or persistent storage. The program stored in the storage media 1230 may include one or more modules (not shown in the figure), each module including a series of instruction operations on the server. Furthermore, the CPU 1222 may be configured to communicate with the storage media 1230 and execute the series of instruction operations in the storage media 1230 on the server.

[0207] Server 140 may also include one or more power supplies 1211, one or more wired or wireless network interfaces 1250, one or more input / output interfaces 1258, and / or one or more operating systems 1241, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0208] The central processing unit 1222 in server 140 can be used to execute the image tag generation method of the present disclosure embodiments.

[0209] This disclosure also provides a computer-readable storage medium for storing program code for executing the image label generation methods of the foregoing embodiments.

[0210] This disclosure also provides a computer program product comprising a computer program. A processor of a computer device reads and executes the computer program, causing the computer device to perform the image tag generation method described above.

[0211] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar terms and are not necessarily used to describe a particular order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “including,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0212] It should be understood that in this disclosure, "at least one item" refers to one or more items, and "more than one item" refers to two or more items. "And / or" is used to describe the relationship between related content, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related content are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0213] It should be understood that in the description of the embodiments disclosed herein, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0214] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0215] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0216] In addition, in the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program with a predetermined function, which works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0217] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0218] It should also be understood that the various implementation methods provided in this disclosure can be combined arbitrarily to achieve different technical effects.

[0219] The above is a detailed description of the embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

1. An image tag generation method, executed by an electronic device, comprising: Acquire a first image and a first image description text used to describe the first image; Keywords are extracted from the description text of the first image and used as keyword tags corresponding to the first image; A target category label corresponding to the first image is determined using a multi-label classification model. The target category label is used to characterize the image category of the first image. The target category label includes at least one of a plurality of preset candidate category labels. The multi-label classification model includes a plurality of label sub-models corresponding to the plurality of candidate category labels. Each label sub-model is used to determine the matching degree between the corresponding candidate category label and the first image. The target category label is determined based on the matching degree between each candidate category label and the first image. The keyword tags and the target category tags are determined as the associated tags for the first image.

2. The image label generation method of claim 1, wherein, The step of extracting keywords from the first image description text includes: The first image description text is filled into a preset prompt template to obtain the first input text. The preset prompt template includes at least one logical analysis prompt word, which is used to indicate the analysis dimension when the first large language model performs semantic analysis on the first image description text. The first input text is input into the first large language model, so that the first large language model performs semantic analysis on the first image description text based on the analysis dimension indicated by the logical analysis prompt words, and obtains the keywords.

3. The image label generation method of claim 2, wherein, The first large language model was trained in the following way: Obtain sample text and decompose the sample text into at least one candidate sample keyword; The sample text and the candidate sample keywords are input into the second language model to determine the sample keywords corresponding to the sample text from the candidate sample keywords. The sample text and the preset prompt template are input into the first large language model to obtain the predicted sample keywords output by the first large language model. Based on the difference between the predicted sample keywords and the sample keywords, the parameters of the first large language model are optimized.

4. The image label generation method according to any one of claims 1 to 3, wherein The step of determining the target category label corresponding to the first image through a multi-label classification model includes: The first image and each of the candidate category labels are respectively formed into image label pairs, wherein a single image label pair includes the first image and a single candidate category label; For each image label pair, the image label pair is input into the label sub-model corresponding to the candidate category label in the image label pair, so as to determine the matching degree between the first image and the candidate category label in the image label pair through the label sub-model; Based on the matching degree between the first image and each of the candidate category labels, the target category label corresponding to the first image is determined from the plurality of candidate category labels.

5. The image label generation method according to claim 4, wherein The multi-label classification model is trained in the following way: Obtain a first training sample set, wherein the first training sample set includes a plurality of first sample images and a sample image label corresponding to each first sample image, and the sample image label includes at least one of the plurality of candidate category labels; The first sample image is combined with each of the candidate category labels to form sample image label pairs, wherein a single sample image label pair includes the first sample image and a single candidate category label; For each sample image label pair, the sample image label pair is input into the label sub-model corresponding to the candidate category label in the sample image label pair, so as to determine the sample matching degree between the first sample image and the candidate category label in the sample image label pair through the label sub-model; Based on the sample matching degree and the sample image label corresponding to the first sample image, a first loss value corresponding to the label sub-model is determined, and the parameters of the label sub-model are optimized based on the first loss value.

6. The image label generation method of claim 5, wherein, The step of determining the first loss value corresponding to the label sub-model based on the sample matching degree and the sample image label corresponding to the first sample image includes: For each candidate category label, the true matching degree between the candidate category label and the first sample image is determined based on the sample image label corresponding to the first sample image; For each of the label sub-models, a first loss value is determined based on the true matching degree between its corresponding candidate category label and each of the first sample images, and the sample matching degree between its corresponding candidate category label and each of the first sample images.

7. The image label generation method according to claim 5 or 6, wherein The optimization of the parameters of the label sub-model based on the first loss value includes: Based on the first loss value, determine the update gradient corresponding to each model parameter of the labeled sub-model; For each model parameter in the labeled sub-model, based on the update gradient and historical gradient corresponding to the model parameter, the update step size corresponding to the model parameter is determined, and the model parameter is updated based on the update step size. The historical gradient is the gradient of the model parameter in the previous iteration.

8. The image tag generation method according to any one of claims 5 to 7, wherein The method further includes: In response to a newly added candidate category label, obtain the original label sub-model corresponding to the newly added candidate category label; Obtain a second training sample set, wherein the second training sample set includes multiple second sample images, and the second sample images are set with a true matching degree between them and the newly added candidate category labels; The predicted matching degree between each second sample image and the newly added candidate category label is determined through the original label sub-model. Based on the predicted matching degree and the true matching degree between each second sample image and the newly added candidate category label, the parameters of the original label sub-model are optimized until the trained original label sub-model is obtained. The trained original label sub-model is added to the multi-label classification model, and the newly added candidate category label is determined as the candidate category label.

9. The image tag generation method according to any one of claims 4 to 8, wherein The step of inputting the image label pair into the label sub-model corresponding to the candidate category label in the image label pair, so as to determine the matching degree between the first image and the candidate category label in the image label pair through the label sub-model, includes: The candidate category labels in the image label pairs are encoded into candidate category label features by a first text encoder, and the first image is encoded into first image features by a first image encoder, wherein the first text encoder and the first image encoder are jointly trained; The candidate category label features and the first image features are concatenated to obtain concatenated features. The concatenated features are then input into a label sub-model corresponding to the candidate category label in the image label pair. The matching degree between the first image and the candidate category label in the image label pair is determined through the label sub-model.

10. The image label generation method of claim 9, wherein, The first text encoder and the first image encoder are trained in the following manner: Obtain a third training sample set, which includes multiple sample image-text pairs, each sample image-text pair including a third sample image and sample description text corresponding to the third sample image; The first text encoder and the first image encoder are used to perform feature encoding on each of the sample description texts and each of the third sample images to obtain multiple sample description text features and multiple sample image features. Determine the predicted matching degree between each of the sample description text features and each of the sample image features, and based on the sample image-text pairs, determine the true matching degree between each of the sample description text features and each of the sample image features; A second loss value is determined based on the difference between the predicted matching degree and the actual matching degree, and the parameters of the first text encoder and the first image encoder are optimized based on the second loss value.

11. The image tag generation method according to any one of claims 1 to 10, wherein After determining the keyword tag and the target category tag as associated tags for the first image, the method includes: In response to an image retrieval request, receive the target image description text; Target keyword tags are determined based on the target image description text, and target association tags are determined based on the target keyword tags. The target association tags are at least one of the association tags corresponding to each first image, and the semantic similarity between the target keyword tags and the target association tags satisfies a preset semantic similarity condition. Based on the target keyword tags and the target association tags, a target image is retrieved from each of the first images and the target image is displayed, wherein the association tags corresponding to the target image include at least a plurality of the target keyword tags and the target association tags.

12. The image tag generation method of claim 11, wherein, The step of determining the target keyword tags based on the target image description text includes: Obtain the object identifier corresponding to the image retrieval request, wherein the object identifier is an identity identifier corresponding to the object that initiated the image retrieval request; Based on the object identifier, at least one historical search text is determined, and the historical search text, the target image description text, and the preset prompt template are input into the first large language model to perform semantic analysis on the historical search text and the target image description text through the first large language model to obtain the target keyword tag.

13. The image label generation method according to claim 11 or 12, wherein The display of the target image includes: The target images are clustered based on the target association tags and the target keyword tags, wherein a single target image corresponds to at least one of the target association tags and the target keyword tags; Display the target image, the target association tags, and the target keyword tags; In response to the selection of the target association tag or the target keyword tag, the target image corresponding to the selected target association tag or target keyword tag is displayed.

14. An image tag generation apparatus, the apparatus comprising: The acquisition unit is used to acquire a first image and a first image description text for describing the first image; The keyword extraction unit is used to extract keywords from the first image description text and use them as keyword tags corresponding to the first image. A category label generation unit is used to determine a target category label corresponding to the first image through a multi-label classification model. The target category label is used to characterize the image category of the first image. The target category label includes at least one of a plurality of preset candidate category labels. The multi-label classification model includes a plurality of label sub-models corresponding to the plurality of candidate category labels. Each label sub-model is used to determine the matching degree between the corresponding candidate category label and the first image. The target category label is determined based on the matching degree between each candidate category label and the first image. A determining unit is used to determine the keyword tag and the target category tag as the associated tags of the first image.

15. An electronic device comprising a memory and a processor, the memory storing a computer program, wherein, When the processor executes the computer program, it implements the image tag generation method according to any one of claims 1 to 13.

16. A computer readable storage medium, said storage medium having stored thereon a computer program, wherein, When the computer program is executed by a processor, it implements the image tag generation method according to any one of claims 1 to 13.

17. A computer program product comprising a computer program that is read and executed by a processor of a computer device, causing the computer device to perform the image tag generation method according to any one of claims 1 to 13.

Images