Image captioning model training method, image captioning method, device, medium, and product
Patent Information
- Application Number
- PCT/CN2025/090471
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-02-18
- Filing Date
- 2025-04-22
- Publication Date
- 2026-08-27
Smart Images
Figure CN2025090471_27082026_PF_FP_ABST
Abstract
Description
Image caption generation model training methods, image caption generation methods, devices, media and products Technical Field
[0001] This application relates to the field of machine learning technology, and in particular to a training method for an image description generation model, an image description generation method, an apparatus, a medium, and a product. Background Technology
[0002] Image description, or image annotation, is a technique that transforms image content into natural language descriptions. Currently, image description is widely used in various scenarios, such as large-scale question answering, visual search, automatic image annotation, content recommendation, and providing assistance to visually impaired individuals. The accuracy of image description directly affects the effectiveness of these applications and user experience, thus possessing significant practical importance.
[0003] Image caption generation mostly employs deep learning-based image captioning models. These models extract basic image features and generate corresponding descriptive sentences based on those features, thus producing the image caption text. Training these models often requires a large training corpus, including training images and their corresponding descriptive sentences. However, current image captioning models are typically trained on English vocabularies, meaning they only support English words. In Chinese scenarios, translation networks are needed to translate Chinese into English, as they cannot directly process Chinese text. This leads to a significant performance drop in image captioning models within Chinese contexts.
[0004] Therefore, how to improve the performance of image description generation models in Chinese scenarios is a technical problem that urgently needs to be solved in this field. Summary of the Invention
[0005] The main purpose of this application is to provide a training method, image description generation method, device, medium and product for an image description generation model, aiming to solve the technical problem of how to improve the performance of the image description generation model in Chinese scenarios.
[0006] To achieve the above objectives, this application provides a method for training an image description generation model, which includes the following steps:
[0007] Obtain the image description generation model to be trained, the original vocabulary and the newly added vocabulary, wherein the newly added vocabulary includes at least one newly added Chinese word and the original vocabulary includes at least one original word;
[0008] For each newly added Chinese word in the newly added vocabulary database, find the target original word in the original vocabulary table whose semantic similarity to the newly added Chinese word is greater than a preset threshold, and insert the newly added Chinese word into the position adjacent to the target original word.
[0009] The expanded vocabulary list is obtained after all newly added Chinese words in the newly added vocabulary database have been inserted into the original vocabulary list.
[0010] The image description generation model is trained based on the expanded vocabulary to obtain the trained image description generation model.
[0011] In one embodiment, prior to the step of training the image description generation model based on the expanded vocabulary, the method further includes:
[0012] Initialize the embedding vectors of each original word in the expanded vocabulary;
[0013] For each newly added Chinese word in the expanded vocabulary, the embedding vector of the target original word adjacent to the newly added Chinese word is determined as the target embedding vector, and the embedding vector of the newly added Chinese word is initialized with the target embedding vector.
[0014] In one embodiment, the step of finding target original words in the original vocabulary list whose semantic similarity to the newly added Chinese words is greater than a preset threshold includes:
[0015] The newly added Chinese vocabulary is input into a pre-trained translation network to obtain the translated vocabulary;
[0016] Compare the semantic similarity between each original word in the original vocabulary list and the translated word, and find the target original word whose semantic similarity is greater than a preset threshold from each original word list.
[0017] Furthermore, to achieve the above objectives, this application also provides an image description generation method, which includes the following steps:
[0018] Obtain the original image and the target image description generation model, and input the original image into the target image description generation model to obtain the description generation result;
[0019] The target image description generation model is an image description generation model trained using the image description generation model training method described above.
[0020] In one embodiment, after the step of inputting the original image into the target image description generation model to obtain the description generation result, the method further includes:
[0021] Retrieve target data that matches the description generated result from a preset text retrieval database.
[0022] In one embodiment, the description generation result is the output data of the output layer of the target image description generation model, and the step of retrieving target data matching the description generation result in a preset text retrieval database includes:
[0023] The description generation result is vectorized to obtain a description vector, and a target index vector matching the description vector is retrieved in a preset text retrieval database.
[0024] The stored data in the preset text retrieval database associated with the target index vector is determined to be the retrieved target data.
[0025] In one embodiment, the description generation result is the input data of the output layer of the target image description generation model, and the step of retrieving target data matching the description generation result in a preset text retrieval database includes:
[0026] Retrieve the target index vector that matches the description generated result from the preset text retrieval database;
[0027] The stored data associated with the target index vector in the preset text retrieval database is determined to be the retrieved target data.
[0028] In addition, to achieve the above objectives, this application also provides an electronic device, the electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the image description generation model training method and / or image description generation method as described above.
[0029] In addition, to achieve the above objectives, this application also provides a readable storage medium, which is a computer-readable storage medium storing a computer program. The computer program is executed by a processor to implement the steps of the image description generation model training method and / or image description generation method as described above.
[0030] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the image description generation model training method and / or image description generation method as described above.
[0031] One or more technical solutions proposed in this application have at least the following technical effects:
[0032] The process involves obtaining an image description generation model to be trained, an original vocabulary, and a newly added vocabulary. The newly added vocabulary includes at least one newly added Chinese word, and the original vocabulary includes at least one original word. For each newly added Chinese word in the newly added vocabulary, a target original word in the original vocabulary with a semantic similarity greater than a preset threshold is found. The newly added Chinese word is then inserted into a position adjacent to the target original word. This process continues until all newly added Chinese words in the newly added vocabulary are inserted into the original vocabulary, resulting in an expanded vocabulary. The image description generation model is then trained based on the expanded vocabulary, yielding a trained image description generation model. Thus, this embodiment inserts at least one newly added Chinese word into the original vocabulary used to train the image description generation model. This means that the model is trained by adding Chinese words to the vocabulary, allowing it to directly process Chinese words without needing a translation network to translate Chinese into English, thereby improving the performance of the image description generation model in Chinese scenes. Furthermore, in this embodiment, the newly added Chinese words are inserted into adjacent positions of the target original words that are semantically similar to them. It is understood that in the model, the word embedding vectors are learned through contextual information. Putting semantically similar words together can make these words more likely to be close to each other in the embedding space, which helps the model to better capture the semantic relationships between words during training, improve the model's semantic understanding ability, and further improve the performance of the image description generation model in Chinese scenarios. At the same time, this structure can reduce the model's dependence on specific words, enabling the model to better handle polysemous words and word variations during training, enhance its adaptability in different contexts, and thus improve the model's generalization ability. Attached Figure Description
[0033] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0034] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0035] Figure 1 is a schematic diagram of the traditional training process for training a BLIP model suitable for Chinese scenarios;
[0036] Figure 2 is a schematic diagram of the traditional inference process of using the BLIP model in a Chinese scenario;
[0037] Figure 3 is a flowchart illustrating the first embodiment of the image description generation model training method of this application;
[0038] Figure 4 is a simplified flowchart of the image description generation model training method of this application;
[0039] Figure 5 is a schematic diagram of the device structure of the image description generation model training device of this application;
[0040] Figure 6 is a schematic diagram of the hardware operating environment of the image description generation model training method device in the embodiments of this application.
[0041] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0042] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0043] Traditional image captioning generation models, such as the BLIP model (Bootstrapping Language-Image Pre-training), are trained on English datasets. This can lead to the model being unable to recognize many words that are not in the vocabulary when performing Chinese tasks, resulting in poor performance in Chinese scenarios.
[0044] Traditionally, in Chinese-language scenarios, a translation network is needed to translate Chinese into English so that the model can indirectly process Chinese. For example, as shown in Figure 1, during the training phase, a translation network is used to translate the Chinese training labels into English, and then the translated English is used as the training labels for the BLIP model. As shown in Figure 2, during the inference phase, the output of the BLIP model needs to be translated back into Chinese using a translation network. This indirect approach to processing Chinese has at least two problems: First, adding a translation network to the model makes it larger and slows down inference. Second, the translation process between English and Chinese leads to the loss of some semantics, thus weakening the model's capabilities and significantly reducing its performance in Chinese-language scenarios.
[0045] Based on this, the main solution of this application is: to obtain an image description generation model to be trained, an original vocabulary, and a newly added vocabulary library, wherein the newly added vocabulary library includes at least one newly added Chinese word, and the original vocabulary includes at least one original word; for each newly added Chinese word in the newly added vocabulary library, a target original word in the original vocabulary with a semantic similarity greater than a preset threshold is found, and the newly added Chinese word is inserted into a position adjacent to the target original word; until all newly added Chinese words in the newly added vocabulary library are inserted into the original vocabulary, an expanded vocabulary is obtained; the image description generation model is trained based on the expanded vocabulary, and the trained image description generation model is obtained. Thus, this application, by inserting at least one newly added Chinese word into the original vocabulary of the image description generation model for training, that is, by adding Chinese words to the vocabulary, trains the image description generation model so that the model can directly process Chinese words without needing to use a translation network to translate Chinese into English, thereby improving the performance of the image description generation model in Chinese scenes. Furthermore, this application inserts newly added Chinese words into adjacent positions of the target original words that are semantically similar to them. It is understood that in the model, the word embedding vectors are learned through contextual information. Putting semantically similar words together can make these words more likely to be close to each other in the embedding space, which helps the model to better capture the semantic relationships between words during training, improve the model's semantic understanding ability, and further improve the performance of the image description generation model in Chinese scenarios. At the same time, this structure can reduce the model's dependence on specific words, enabling the model to better handle polysemous words and word variations during training, enhance its adaptability in different contexts, and thus improve the model's generalization ability.
[0046] It should be noted that the execution subject of the image description generation model training method in this application can be a computing service device with data processing, model communication, and program execution functions, such as a server, tablet computer, personal computer, mobile phone, etc., or an electronic device capable of realizing the above functions, such as a robot. The embodiments of the image description generation model training method in this application do not impose specific limitations in this regard.
[0047] Based on this, this application proposes a first embodiment of an image description generation model training method. Referring to FIG3, the image description generation model training method includes the following steps S10 to S40:
[0048] Step S10: Obtain the image description generation model to be trained, the original vocabulary and the newly added vocabulary, wherein the newly added vocabulary includes at least one newly added Chinese word and the original vocabulary includes at least one original word.
[0049] This image description generation model is used to generate image descriptions, that is, a model that converts images into text descriptions. Specifically, it can be a model built based on a deep learning architecture, such as the BLIP model or the CLIP (Contrastive Language-Image Pre-training) model. This embodiment does not impose any specific limitations on this. For example, the various embodiments of this application are described and illustrated using the BLIP model as the image description generation model.
[0050] This raw vocabulary is the vocabulary currently used to train the BLIP model, such as the WordPiece vocabulary, which contains the set of words used by the model during the training phase. The raw vocabulary typically consists of common English words, sub-words, and some special markers. For example, the raw vocabulary might include common words like "man," "woman," "tree," and "car," as well as sub-word units like "##ing" and "##ed."
[0051] The newly added vocabulary refers to a newly added vocabulary database containing at least one new Chinese word, which must be a word from the Chinese language. The new Chinese words in this newly added vocabulary database can be derived from Chinese lexicons, such as the lexicon used to train the Chinese BERT (Bidirectional Encoder Representations from Transformers) model, specialized terminology databases, user-defined lexicons (e.g., in game applications, "Sun Wukong," "Chang'e," "Peach Garden," "Dragon Palace," etc.), dynamically crawled emerging words, etc. This embodiment does not impose specific limitations on this.
[0052] Step S20: For each newly added Chinese word in the newly added vocabulary database, find the target original word in the original vocabulary table whose semantic similarity with the newly added Chinese word is greater than a preset threshold, and insert the newly added Chinese word into the position adjacent to the target original word;
[0053] It should be noted that if the target original word is not found in the original vocabulary, meaning there is no target original word in the original vocabulary with a semantic similarity greater than a preset threshold between the newly added Chinese word and the target original word, the newly added Chinese word can be inserted at any position in the original vocabulary, such as at the end of the original vocabulary. If there are multiple original words in the original vocabulary with a semantic similarity greater than the preset threshold between them and the newly added Chinese word, the original word with the highest semantic similarity can be selected as the target original word. This groups words with higher semantic similarity together, which helps the model better handle semantic similarity tasks, such as word sense similarity calculation and analogy reasoning.
[0054] After finding the target original word in the original vocabulary list, the new Chinese word is inserted into the position adjacent to the target original word. Specifically, the position adjacent to the target original word can be one position before or after the target original word.
[0055] Step S30: After all the newly added Chinese words in the new vocabulary database have been inserted into the original vocabulary list, an expanded vocabulary list is obtained.
[0056] After inserting all newly added Chinese words from the new vocabulary into the original vocabulary list according to semantic order, an expanded vocabulary list is obtained. In other words, the expanded vocabulary list is the vocabulary list obtained after inserting all newly added Chinese words into the original vocabulary list.
[0057] Step S40: Train the image description generation model based on the expanded vocabulary to obtain the trained image description generation model.
[0058] After obtaining the expanded vocabulary, the BLIP model is trained based on the expanded vocabulary. If the preset training termination condition is met during the training process, the trained BLIP model is obtained. If the preset training termination condition is not met, the model parameters of the BLIP model are iteratively optimized until the preset training termination condition is met.
[0059] The training termination condition can be a pre-set condition, such as reaching a predetermined number of iterations, the loss function value falling below a predetermined threshold, exhausting computing resources, reaching a time limit, or the accuracy reaching a predetermined threshold. This embodiment does not impose specific restrictions on this.
[0060] It is understandable that training a BLIP model requires images as input and corresponding text descriptions as labels. Therefore, an expanded vocabulary and a pre-defined image-text pair dataset can be used to train the BLIP model. The image-text pair dataset includes multiple images and their corresponding text descriptions. Specifically, publicly available datasets such as the COCO (Common Objects in Context) dataset can be used, or datasets that can be prepared in advance based on actual needs. This embodiment does not impose any specific limitations on this.
[0061] This embodiment obtains an image description generation model to be trained, an original vocabulary, and a newly added vocabulary library. The newly added vocabulary library includes at least one newly added Chinese word, and the original vocabulary includes at least one original word. For each newly added Chinese word in the newly added vocabulary library, a target original word in the original vocabulary with a semantic similarity greater than a preset threshold is found, and the newly added Chinese word is inserted into a position adjacent to the target original word. This process continues until all newly added Chinese words in the newly added vocabulary library are inserted into the original vocabulary, resulting in an expanded vocabulary. The image description generation model is then trained based on the expanded vocabulary, yielding a trained image description generation model. Thus, this embodiment inserts at least one newly added Chinese word into the original vocabulary for training the image description generation model. This means that the image description generation model is trained by adding Chinese words to the vocabulary, allowing the model to directly process Chinese words without needing a translation network to translate Chinese into English, thereby improving the performance of the image description generation model in Chinese scenes. Furthermore, in this embodiment, newly added Chinese words are inserted into adjacent positions of the target original words that are semantically similar to them. It can be understood that in the model, the word embedding vectors are learned through contextual information. Putting semantically similar words together makes these words more likely to be close to each other in the embedding space, which helps the model to better capture the semantic relationships between words during training, improve the model's semantic understanding ability, and further improve the performance of the image description generation model in Chinese scenarios. At the same time, this structure can reduce the model's dependence on specific words, enabling the model to better handle polysemous words and word variations during training, enhance its adaptability in different contexts, and thus improve the model's generalization ability.
[0062] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Furthermore, before the step of training the image description generation model based on the expanded vocabulary, the method further includes:
[0063] Step A10: Initialize the embedding vectors of each original word in the expanded vocabulary;
[0064] It should be noted that the embedding vector refers to a high-dimensional vector that maps a word to its embedding layer in the BLIP model. For the original words in the expanded vocabulary, i.e., the words initially in the original vocabulary, they can be initialized using the same method as the current method for initializing BLIP model embedding vectors. For example, before training the BLIP model, a pre-trained text encoder (such as BERT or its variants) is loaded. The BLIP model uses the word embedding layer in the pre-trained text encoder to initialize the embedding vectors of each original word in the original vocabulary.
[0065] Step A20: For each newly added Chinese word in the expanded vocabulary, determine the embedding vector of the target original word adjacent to the newly added Chinese word as the target embedding vector, and initialize the embedding vector of the newly added Chinese word with the target embedding vector.
[0066] For newly added Chinese words to the original vocabulary, their embedding vectors are vectorized using the embedding vectors of the target original words adjacent to them (i.e., the target embedding vectors). Specifically, for each new Chinese word, the distance between its embedding vector and the target embedding vector is less than a certain threshold. For example, the embedding vector of the new Chinese word can be initialized with a weighted average of the target embedding vectors or a nearby value, so that the two words are relatively close in the embedding space. This reduces the number of iterations during model training, accelerates model convergence, and improves model training efficiency.
[0067] Furthermore, the dimension of the output vector of the final output layer of the BLIP model is modified to the table dimension of the expanded vocabulary. The table dimension of the expanded vocabulary refers to the total number of words in the expanded vocabulary. For example, if the expanded vocabulary contains m words, the dimension of the output vector of the final output layer of the BLIP model is modified to m dimensions.
[0068] Furthermore, for the initialization of the output vector weights of the final output layer of the BLIP model, the mean-filling method can be adopted. That is, the mean of the initial weights of all original words in the original vocabulary in the output vector is used to fill the initial weights of the newly added Chinese words in the output vector, so as to make the model parameters of the BLIP model evenly distributed, reduce the number of iterations during model training, and improve the training efficiency of the model.
[0069] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to the first and second embodiments described above can be referred to the above description and will not be repeated hereafter. Based on this, the step of finding target original words in the original vocabulary list whose semantic similarity to the newly added Chinese words is greater than a preset threshold includes:
[0070] Step B10: Input the newly added Chinese vocabulary into the pre-trained translation network to obtain the translated vocabulary;
[0071] It should be noted that this translation network is used to translate newly added Chinese words into translation words in the same language as the original words in the original vocabulary list. If the original vocabulary list contains original words belonging to multiple languages, the translation network will translate the newly added Chinese words into the translation word of the language with the highest percentage, where the language with the highest percentage is the language to which each original word in the original vocabulary list belongs. For example, assuming that all or more than half of the original words in the original vocabulary list belong to English, the translation network will translate the newly added Chinese words into English.
[0072] Step B20: Compare the semantic similarity between each original word in the original vocabulary list and the translated word, and find the target original word whose semantic similarity is greater than a preset threshold from each original word.
[0073] It is understandable that calculating and obtaining semantic similarity between words in the same language is more convenient and simpler than calculating and obtaining semantic similarity between words in different languages. Based on this, in this embodiment, a translation network is used to translate the target original vocabulary to obtain translated vocabulary. This allows for a comparison of semantic similarity with translated vocabulary in the same language as most of the original vocabulary, thereby finding the target original vocabulary and improving the efficiency of target original vocabulary retrieval.
[0074] For example, to help understand the technical concept or principle of the image description generation model training method after combining this embodiment with the first and second embodiments, a specific embodiment is listed below. In this specific embodiment, please refer to Figure 4. The text encoder and decoder of the BLIP model are modified. First, the original vocabulary of the English BLIP model text to token (the original token vocabulary shown in Figure 4) is modified. Specifically, Chinese words are added to the original vocabulary to obtain an expanded vocabulary (the expansion of this table, adding Chinese words, is shown in Figure 4). This part of the Chinese vocabulary is taken from the vocabulary of the Chinese BERT model and covers all Chinese words.
[0075] After adjusting the original vocabulary, the weights of the corresponding embedding layers in the model are modified. First, the embedding dimension of the original model is changed to the table dimension of the expanded vocabulary. Second, the weights of this expanded dimension are initialized. In this specific implementation, the initialization strategy uses mean padding, where the mean is derived from the average of the embedding weights in the same layer. Finally, the dimension of the final output layer of the model's text decoder is modified, also changing it to the table dimension of the expanded vocabulary, and the weights are initialized using mean padding as well.
[0076] After initializing the weights of the Embedding layer (the initial layer shown in Figure 4) and the output layer in the BLIP model, the BLIP model is trained with an expanded vocabulary to obtain the trained BLIP model.
[0077] It should be noted that the above examples are only used to help understand this application and do not constitute a limitation on the training method of the image description generation model of this application. Any simple transformations based on this technical concept are within the protection scope of this application.
[0078] Furthermore, this application also proposes an image description generation method, which includes the following steps S100:
[0079] Step S100: Obtain the original image and the target image description generation model to be generated; input the original image into the target image description generation model to obtain the description generation result.
[0080] The target image description generation model is an image description generation model trained using the image description generation model training method described in any of the above embodiments.
[0081] The execution entity of each embodiment of the image description generation method can be a computing service device with data processing, model communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone; or an electronic device capable of performing the above functions, such as AR (Augmented Reality) glasses, VR (Virtual Reality) glasses, AR headsets, VR headsets, smart glasses, or headphones. This embodiment does not impose specific limitations on this. Furthermore, the execution entity of the image description generation method and the image description generation model training method of this application can be the same or different, and no specific limitations are imposed here.
[0082] It is easy to understand that after obtaining the original image to be described, the original image is input into the target image description generation model to obtain the description generation result, which can be the output of the target image description generation model.
[0083] In one possible implementation, after the step of inputting the original image into the target image description generation model to obtain the description generation result, the method further includes:
[0084] Step S200: Retrieve target data that matches the description generation result from a preset text retrieval database.
[0085] The preset text retrieval database can be a database pre-set by relevant personnel, specifically a RAG (Retrieval-Augmented Generation) database. A RAG database is a vector database specifically designed for storing and querying high-dimensional vector data. It converts text data into vectors (embedding) and stores these vectors, enabling efficient retrieval of the vector closest to the query vector, thereby finding relevant text content.
[0086] Traditional RAG image processing uses OCR or large model techniques to extract key information from the image for text encoding. However, when the image lacks text information or its content cannot be understood by the large model, image input into RAG becomes ineffective. Therefore, in a preferred implementation, the preset text retrieval database is a RAG database. This allows a fine-tuned BLIP model to convert the image into a corresponding text description, enabling subsequent RAG retrieval based on the text. This achieves image encoding into RAG format and low-cost image retrieval into the RAG database.
[0087] Considering the rapid increase in the variety of smart AR+AI (Artificial Intelligence) glasses, it has become increasingly easy to capture images with smart devices. Simultaneously, with the emergence of large-scale visual models, enabling these models to better understand the content of images has become crucial. Generally, large-scale visual models cannot achieve detailed understanding of images in specific scenarios (such as game scenarios). Achieving detailed understanding requires training and fine-tuning the large-scale visual model, but the cost of fine-tuning is very high. Therefore, this embodiment uses a target image description generation model to generate text descriptions of images, and then performs searches in a text retrieval database based on these text descriptions. This allows images to be retrieved within text-based data, realizing an end-to-end large-scale model + text retrieval solution. This approach improves the visual capabilities of large-scale models at a low cost, better adapting them to specific scenarios.
[0088] In one possible implementation, the description generation result is the output data of the output layer of the target image description generation model, and the step of retrieving target data matching the description generation result in a preset text retrieval database includes:
[0089] Step S201: The description generation result is vectorized to obtain a description vector, and a target index vector matching the description vector is retrieved in a preset text retrieval database.
[0090] It should be noted that this output layer specifically refers to the last layer of the target image description generation model. Its purpose is to transform the model's internal feature representation into the final prediction result, enabling the model to directly output an interpretation or prediction of the input data. This description generation result is the output data of the output layer of the target image description generation model, which is also the final output of the target image description generation model.
[0091] If the description generation result is the output data of the output layer of the target image description generation model, and the index of the preset text retrieval database is stored in vector form, then the description generation result is vectorized so that the target index vector can be matched in the preset retrieval database through vector matching.
[0092] Vectorization refers to the process of converting data into numerical vectors. The specific processing method for vectorization can be preset. For example, data vectorization can be accomplished through embedding (such as Text2Vec embedding networks). Embedding processing usually refers to converting data from its original form into a low-dimensional, continuous vector representation that can capture the inherent characteristics and structure of the data.
[0093] Step S202: Determine that the stored data in the preset text retrieval database associated with the target index vector is the retrieved target data.
[0094] Specifically, the target data can be the original text fragments corresponding to the target index vector stored in a preset text retrieval database.
[0095] In one possible implementation, the description generation result is the input data of the output layer of the target image description generation model, and the step of retrieving target data matching the description generation result in a preset text retrieval database includes:
[0096] Step S203: Retrieve a target index vector that matches the description generation result from a preset text retrieval database;
[0097] It should be noted that the target image description generation result is the input data of the output layer of the target image description generation model, that is, the output of the penultimate layer of the target image description generation model, such as the output of the LM layer (Language Modeling Layer) of the BLIP model. In this case, it can be understood that the output of the second-to-last layer of the model is essentially vector-mode data. Therefore, when the index of the preset text retrieval database is stored in vector format, there is no need to vectorize the description generation result; the target index vector can be directly obtained by matching the target index vector in the preset retrieval database based on the description generation result.
[0098] Step S204: Determine that the stored data in the preset text retrieval database associated with the target index vector is the retrieved target data.
[0099] Similarly, the target data can specifically be the original text fragments corresponding to the target index vector stored in a preset text retrieval database.
[0100] Furthermore, this application also proposes an image description generation model training device. Referring to FIG5, the image description generation model training device includes:
[0101] The acquisition module 10 is used to acquire the image description generation model to be trained, the original vocabulary and the newly added vocabulary, wherein the newly added vocabulary includes at least one newly added Chinese word and the original vocabulary includes at least one original word.
[0102] The insertion module 20 is used to find, for each newly added Chinese word in the newly added vocabulary library, a target original word in the original vocabulary list whose semantic similarity to the newly added Chinese word is greater than a preset threshold, and insert the newly added Chinese word into a position adjacent to the target original word.
[0103] The insertion module 20 is also used to obtain an expanded vocabulary list after all the newly added Chinese words in the new vocabulary list have been inserted into the original vocabulary list;
[0104] Training module 30 is used to train the image description generation model based on the extended vocabulary to obtain the trained image description generation model.
[0105] Furthermore, embodiments of this application also propose an electronic device, the electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the image description generation model training method and / or image description generation method as described above.
[0106] Referring to Figure 6, a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of this application is shown. The electronic device in the embodiments of this application may also include, but is not limited to, mobile terminals such as headphones, AR glasses, VR glasses, AR headsets, VR headsets, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), etc., as well as fixed terminals such as digital TVs, desktop computers, etc. The electronic device shown in Figure 6 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.
[0107] As shown in Figure 6, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. While electronic devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.
[0108] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a model via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.
[0109] The electronic device provided in this application, employing the image description generation model training method and / or image description generation method described in the above embodiments, can solve the technical problem of how to improve the performance of the image description generation model in Chinese scenarios. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the image description generation model training method and / or image description generation method described in the above embodiments, and other technical features of this electronic device are the same as those disclosed in the previous embodiment method, and will not be repeated here.
[0110] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.
[0111] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0112] In addition, to achieve the above objectives, embodiments of this application also provide a readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the image description generation model training method and / or image description generation method in the above embodiments.
[0113] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.
[0114] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.
[0115] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by an electronic device, the electronic device causes the following to occur: It acquires an image description generation model to be trained, an original vocabulary, and a newly added vocabulary library, wherein the newly added vocabulary library includes at least one newly added Chinese word, and the original vocabulary includes at least one original word; for each newly added Chinese word in the newly added vocabulary library, it searches for a target original word in the original vocabulary library whose semantic similarity to the newly added Chinese word is greater than a preset threshold, and inserts the newly added Chinese word into a position adjacent to the target original word; until all newly added Chinese words in the newly added vocabulary library are inserted into the original vocabulary, an expanded vocabulary is obtained; and the image description generation model is trained based on the expanded vocabulary, thereby obtaining the trained image description generation model.
[0116] Computer program code for performing the operations of this application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, as well as conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of model—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0117] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0118] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the modules themselves.
[0119] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described image description generation model training method and / or image description generation method, thereby solving the technical problem of how to improve the performance of the image description generation model in Chinese scenarios. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the image description generation model training method and / or image description generation method provided in the above embodiments, and will not be repeated here.
[0120] Furthermore, embodiments of this application also propose a computer program product, including a computer program that, when executed by a processor, implements the steps of the image description generation model training method and / or image description generation method as described above.
[0121] The specific implementation of the computer program product in this application is basically the same as the various embodiments of the above-described image description generation model training and / or image description generation method, and will not be repeated here.
[0122] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.
[0123] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0124] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software sensor. This computer software sensor is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or model device, etc.) to execute the methods described in the various embodiments of this application.
[0125] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. A method for training an image description generation model, characterized in that, The image description generation model training method comprises the following steps: obtaining an image description generation model to be trained, an original vocabulary table and an added vocabulary library, wherein the added vocabulary library comprises at least one added Chinese vocabulary, and the original vocabulary table comprises at least one original vocabulary; for each added Chinese vocabulary in the added vocabulary library, finding a target original vocabulary in the original vocabulary table that has a semantic similarity greater than a preset threshold with the added Chinese vocabulary, and inserting the added Chinese vocabulary into a position adjacent to the target original vocabulary; until all added Chinese vocabularies in the added vocabulary library are inserted into the original vocabulary table to obtain an expanded vocabulary table; training the image description generation model based on the expanded vocabulary table to obtain a trained image description generation model. 2.The image description generation model training method of claim 1, wherein, Before the step of training the image description generation model based on the expanded vocabulary table, the method further comprises: initializing the embedding vectors of each original vocabulary in the expanded vocabulary table; for each added Chinese vocabulary in the expanded vocabulary table, determining the embedding vector of the target original vocabulary adjacent to the added Chinese vocabulary as a target embedding vector, and initializing the embedding vector of the added Chinese vocabulary with the target embedding vector. 3.The image description generation model training method of claim 1, wherein, The step of finding a target original vocabulary in the original vocabulary table that has a semantic similarity greater than a preset threshold with the added Chinese vocabulary comprises: inputting the added Chinese vocabulary into a pre-trained translation network to obtain a translated vocabulary; comparing the semantic similarity between each original vocabulary in the original vocabulary table and the translated vocabulary, and finding a target original vocabulary from each original vocabulary that has a semantic similarity greater than a preset threshold.
4. An image description generation method characterized by, The image description generation method comprises the following steps: obtaining an original image and a target image description generation model to be generated, and inputting the original image into the target image description generation model to obtain a description generation result; wherein the target image description generation model is an image description generation model trained by the image description generation model training method of any one of claims 1 to 3.
5. The image description generation method of claim 4, wherein, After the step of inputting the original image into the target image description generation model to obtain a description generation result, the method further comprises: retrieving target data matching the description generation result in a preset text retrieval database.
6. The image description generation method of claim 5, wherein, The description generation result is the output data of the output layer of the target image description generation model, and the step of retrieving target data matching the description generation result in a preset text retrieval database comprises: vectorizing the description generation result to obtain a description vector, and retrieving a target index vector matching the description vector in a preset text retrieval database; determining the storage data associated with the target index vector in the preset text retrieval database as the retrieved target data.
7. The image description generation method of claim 5, wherein, The description generation result is the input data of the output layer of the target image description generation model, and the step of retrieving targetdata matching the description generation result in a preset text retrieval database comprises: retrieving a target index vector matching the description generation result in a preset text retrieval database; determining stored data associated with the target index vector in the preset text retrieval database as the retrieved target data.
8. An electronic device, comprising: The electronic device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the method according to any one of claims 1 to 7.
9. A readable storage medium, characterized by, The readable storage medium is a computer readable storage medium, and the readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the method according to any one of claims 1 to 7.
10. A computer program product, characterised in that, The computer program product comprises a computer program, and the computer program is executed by a processor to implement the steps of the method according to any one of claims 1 to 7.