Data processing method, computer device, and readable storage medium

Abstract

The present application relates to a data processing method and apparatus, a computer device, and a readable storage medium. The method comprises: acquiring an image-text knowledge dataset, the image-text knowledge dataset comprising a plurality of groups of data, each group of data comprising an image, a knowledge text corresponding to the image, and a query question corresponding to the knowledge text; for each group of data, acquiring a global description of a target image contained in current data, performing target detection on the target image on the basis of a keyword contained in the global description to obtain a target detection bounding box, acquiring a local image set on the basis of the target detection bounding box, and acquiring a local description of each local image in the local image set to obtain a local description set, the global description, the local image set, the local description set and the current data constituting an image-text matching pair; and constructing positive and negative sample pairs on the basis of an image-text matching pair corresponding to each group of data, and training a multimodal model on the basis of the positive and negative sample pairs.

Description

Data processing methods, computer equipment and readable storage media

[0001] Related applications

[0002] This application claims priority to Chinese patent application filed on December 5, 2024, application number 202411777279.4, entitled "Data Processing Method, Computer Equipment and Readable Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of machine learning technology, and in particular to a data processing method, apparatus, computer device, computer-readable storage medium, and computer program product. Background Technology

[0004] In recent years, with the development of deep learning technology, deep learning-based language models have gradually become mainstream. Among them, multimodal models allow users to ask questions using a combination of text and images. However, the accuracy of responses output by current multimodal models is not high. Summary of the Invention

[0005] Therefore, it is necessary to provide a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product to address the aforementioned technical problems.

[0006] Firstly, this application provides a data processing method, including:

[0007] Obtain the image and text knowledge dataset, which includes multiple sets of data. Each set of data includes an image, the knowledge text corresponding to the image, and the query question corresponding to the knowledge text.

[0008] For each set of data, obtain a global description of the target image contained in the current data, perform target detection on the target image based on the keywords contained in the global description to obtain target detection boxes, and obtain a local image set based on the target detection boxes; obtain a local description of each local image in the local image set to obtain a local description set; the global description, local image set, local description set and the current data constitute an image-text matching pair;

[0009] Based on the image-text matching pairs corresponding to each set of data, positive and negative sample pairs are constructed; positive and negative sample pairs are used to train the multimodal model.

[0010] In one embodiment, obtaining a local image set based on the target detection bounding box includes:

[0011] Based on the target detection bounding box, the target image is segmented to obtain a local image of the target;

[0012] Based on the pre-set image segmentation rules, the number of horizontal and vertical segments is obtained, and the target image is segmented into at least one local image region based on the number of horizontal and vertical segments.

[0013] A local image set is constructed based on the target local image and the region local image.

[0014] In one embodiment, positive and negative sample pairs are constructed based on the image-text matching pairs corresponding to each set of data, including:

[0015] For each set of data corresponding to the image-text matching pair, obtain the target knowledge text and target local description set contained in the current image-text matching pair; encode the target knowledge text to obtain the knowledge text vector; encode each local description in the target local description set to obtain the local text vector of each local description;

[0016] Based on the knowledge text vector and the local text vector of each local description, the current image-text match is split into positive and negative sample groups;

[0017] Positive and negative sample pairs are constructed based on the positive and negative sample groups obtained by splitting the image and text matching pairs corresponding to each set of data.

[0018] In one embodiment, based on the knowledge text vector and the local text vector of each local description, the current image-text match is split into positive and negative sample groups, including:

[0019] For each local description, calculate the cosine distance between the knowledge text vector and the local text vector of the current local description. If the cosine distance is greater than or equal to a preset cosine distance threshold, the current local description is taken as a positive sample of the local description; if the cosine distance is less than the preset cosine distance threshold, the current local description is taken as a negative sample of the local description. All positive samples of local descriptions constitute the positive sample set of local descriptions, and all negative samples of local descriptions constitute the negative sample set of local descriptions.

[0020] A positive sample group is constructed based on the local description positive sample set, the local image corresponding to each local description in the local description positive sample set, the corresponding global description, and the corresponding data.

[0021] A negative sample group is constructed based on the local description negative sample set and the local image corresponding to each local description in the local description negative sample set.

[0022] In one embodiment, positive and negative sample pairs are constructed based on the positive and negative sample groups obtained by splitting the image-text matching pairs corresponding to each group of data, including:

[0023] For each set of data corresponding to the image and text matching pair, obtain the target positive and negative sample groups obtained by splitting the current image and text matching pair, and take the positive sample group in the target positive and negative sample group as a positive sample pair;

[0024] Randomly select target data from all data in the image-text knowledge dataset except for the data corresponding to the current image-text matching pair. Construct negative sample pairs based on the negative sample group in the target positive and negative sample group, the global description of the image contained in the target data, and the target data.

[0025] In one embodiment, training the multimodal model based on positive and negative sample pairs includes:

[0026] Based on positive and negative sample pairs, comparative learning is performed on the multimodal model to be trained to obtain a basic multimodal model;

[0027] Based on the image-text matching pairs and preset instruction templates corresponding to each set of data, an instruction fine-tuning dataset is generated.

[0028] Based on the instruction fine-tuning dataset, the basic multimodal model is trained by instruction fine-tuning to obtain the trained multimodal model.

[0029] In one embodiment, the method further includes:

[0030] Obtain the data to be imported, which includes the images to be imported, the corresponding knowledge text for the images to be imported, and the corresponding query questions for the knowledge text to be imported.

[0031] Obtain a global description of the imported images as the global description of the imported images. Perform target detection on the imported images based on the keywords contained in the global description of the imported images. Obtain a set of local images in the imported images based on the detection results. Obtain a local description of each local image in the set of local images in the imported images to obtain a set of local descriptions in the imported images.

[0032] The global description of the database is encoded to obtain the global description vector; the knowledge text of the database is encoded to obtain the knowledge text vector; and each local description in the local description set of the database is encoded to obtain the local text vector.

[0033] Calculate the cosine distance between the knowledge text vector and each local text vector in the database, and sort the local descriptions in the local description set in the database according to the order of cosine distance from smallest to largest, and obtain the target local descriptions at the top of the list.

[0034] Extract the local image embedding vector of the local image corresponding to the local description of each target; extract the global image embedding vector of the images in the database;

[0035] Based on the global description vector, target local description, local image embedding vector, global image embedding vector, and knowledge text vector, knowledge groups are constructed and stored in the knowledge base.

[0036] In one embodiment, the method further includes:

[0037] Receive search requests, which include query images and query questions;

[0038] Obtain a global description of the queried image;

[0039] Based on the global description of the query image, at least one target knowledge group whose similarity meets the preset conditions is retrieved from the knowledge base;

[0040] Randomly select one image local description instruction from the preset instruction set for image local description, and generate multiple local descriptions based on the image local description instruction, global description, query question, and knowledge text in the target knowledge group;

[0041] The extended query problem is obtained by expanding the query problem based on multiple local descriptions;

[0042] Based on the query question and extended questions, a search is conducted in at least one target knowledge group to obtain the final knowledge group, and a search response is generated based on the final knowledge group.

[0043] Secondly, this application also provides a data processing apparatus, comprising:

[0044] The acquisition module is configured to acquire a graph and text knowledge dataset, which includes multiple sets of data. Each set of data includes an image, the knowledge text corresponding to the image, and the query question corresponding to the knowledge text.

[0045] The construction module is configured to, for each set of data, obtain a global description of the target image contained in the current data, perform target detection on the target image based on the keywords contained in the global description to obtain a target detection box, obtain a local image set based on the target detection box, obtain a local description set for each local image in the local image set, and then obtain a local description set. The global description, the local image set, the local description set, and the current data constitute an image-text matching pair.

[0046] The training module is configured to construct positive and negative sample pairs based on the image-text matching pairs corresponding to each set of data; and to train the multimodal model based on the positive and negative sample pairs.

[0047] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0048] Obtain the image and text knowledge dataset, which includes multiple sets of data. Each set of data includes an image, the knowledge text corresponding to the image, and the query question corresponding to the knowledge text.

[0049] For each set of data, obtain a global description of the target image contained in the current data, perform target detection on the target image based on the keywords contained in the global description to obtain target detection boxes, and obtain a local image set based on the target detection boxes; obtain a local description of each local image in the local image set to obtain a local description set; the global description, local image set, local description set and the current data constitute an image-text matching pair;

[0050] Based on the image-text matching pairs corresponding to each set of data, positive and negative sample pairs are constructed; the multimodal model is trained based on the positive and negative sample pairs.

[0051] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0052] Obtain the image and text knowledge dataset, which includes multiple sets of data. Each set of data includes an image, the knowledge text corresponding to the image, and the query question corresponding to the knowledge text.

[0053] For each set of data, obtain a global description of the target image contained in the current data, perform target detection on the target image based on the keywords contained in the global description to obtain target detection boxes, and obtain a local image set based on the target detection boxes; obtain a local description of each local image in the local image set to obtain a local description set; the global description, local image set, local description set and the current data constitute an image-text matching pair;

[0054] Based on the image-text matching pairs corresponding to each set of data, positive and negative sample pairs are constructed; the multimodal model is trained based on the positive and negative sample pairs.

[0055] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0056] Obtain the image and text knowledge dataset, which includes multiple sets of data. Each set of data includes an image, the knowledge text corresponding to the image, and the query question corresponding to the knowledge text.

[0057] For each set of data, obtain a global description of the target image contained in the current data, perform target detection on the target image based on the keywords contained in the global description to obtain target detection boxes, and obtain a local image set based on the target detection boxes; obtain a local description of each local image in the local image set to obtain a local description set; the global description, local image set, local description set and the current data constitute an image-text matching pair;

[0058] Based on the image-text matching pairs corresponding to each set of data, positive and negative sample pairs are constructed; the multimodal model is trained based on the positive and negative sample pairs.

[0059] Details of one or more embodiments of this application are set forth in the following drawings and description. Other features, objects, and advantages of this application will become apparent from the specification, drawings, and claims. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the disclosed drawings without creative effort.

[0061] Figure 1 is a schematic flowchart of a data processing method in one or more embodiments;

[0062] Figure 2 is an example diagram of data from one or more embodiments / some embodiments;

[0063] Figure 3 is an example diagram showing the generation of a global description of one or more embodiments / some embodiments;

[0064] Figure 4 is an example diagram of a partial image of a region in one or more embodiments / some embodiments;

[0065] Figure 5 is a flowchart illustrating the data processing method of one or more embodiments / some embodiments;

[0066] Figure 6 is a data entry flowchart for one or more embodiments / some embodiments;

[0067] Figure 7 is a retrieval flowchart of one or more embodiments / some embodiments;

[0068] Figure 8 is an internal structure diagram of a computer device according to one or more embodiments / some embodiments. Detailed Implementation

[0069] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0070] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0071] In one embodiment, as shown in Figure 1, a data processing method is provided. This embodiment illustrates the method applied to a terminal, but it is understood that the method can also be applied to a server, or to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0072] Step 102: Obtain the image and text knowledge dataset. The image and text knowledge dataset includes multiple sets of data. Each set of data includes an image, the knowledge text corresponding to the image, and the query question corresponding to the knowledge text.

[0073] Optionally, a graph-text knowledge dataset for a specific professional field can be collected. This graph-text knowledge dataset includes multiple sets of data, each set of data including at least one image in the professional field, the knowledge text corresponding to the image, and the query question corresponding to the knowledge text.

[0074] In order to obtain massive image and text knowledge datasets, data is usually acquired from multiple channels. Therefore, images may contain multiple elements unrelated to the text content. As shown in Figure 2, images from a set of data, such as average fuel consumption and speedometer readings, are displayed. When the warning light is indicated by the arrow in Figure 2, the information it indicates is: Tire pressure warning (indicator light is amber), meaning that the pressure of a certain tire is out of range. If a malfunction of the Tire Pressure Monitoring System (TPMS) is detected, the corresponding indicator light will flash.

[0075] Step 104: For each set of data, obtain a global description of the target image contained in the current data, perform target detection on the target image based on the keywords contained in the global description to obtain a target detection box, and obtain a local image set based on the target detection box; obtain a local description of each local image in the local image set to obtain a local description set; the global description, the local image set, the local description set, and the current data constitute an image-text matching pair.

[0076] Optionally, a native visual-text multimodal model M to be fine-tuned can be prepared in advance, such as BLIPv2, GLM-4V-9B, etc. For the sake of illustration, the visual-text multimodal model mentioned in the embodiments of this application can be simply referred to as a multimodal model. For each set of data, a global description of the target image contained in the current data can be obtained through the multimodal model M.

[0077] Specifically, for each set of data, a multimodal model M can be used to process the target images contained in the current data to generate a global description T. g When processing a target image using a multimodal model M, the cue word P can be used. g P g The expression "Describe this graph" can be represented by the following mathematical expression, T. gThe generation process of T: g =M(I,P g )

[0078] For example, referring to Figure 3, using the prompt "Describe this image," after processing the image shown in Figure 3 using the multimodal model M, the generated global description is: "The image shows the hood of a car open. In the center of the image is a silver engine with a silver logo that reads 'TURBO.' In the upper right corner of the image, there is a red container containing liquid, possibly coolant or brake fluid. In addition, there are other car parts and pipes, but their specific functions are unclear."

[0079] Based on the cited content, here are some potential target terms:

[0080] 1. "TURBO": This is the mark of a car turbocharger, which is usually used to improve the intake pressure of the engine and increase the engine's power output.

[0081] 2. "Engine oil": This is an essential lubricant for the operation of a car engine, which helps reduce friction and wear between engine parts.

[0082] 3. "Battery": This is a key component that powers the car's starting and electrical systems.

[0083] 4. "Alternator": It is responsible for charging the car's electrical system while the vehicle is in motion. 5. "Air Conditioning Compressor": If the image shows the air conditioning system, the air conditioning compressor is the component used to compress refrigerant gas, causing it to circulate within the system and cool the air inside the vehicle.

[0084] 6. "Radiator": If the image shows the cooling system, then the radiator is a component used to transfer the heat generated by the engine to the surrounding air, thereby maintaining the engine's normal operating temperature.

[0085] 7. "Brake fluid": This is the fluid in the braking system used to transmit the force when the driver presses the brake pedal, causing the wheels to slow down or stop.

[0086] 8. "Transmission fluid": This is the lubricant for automatic transmissions, which helps keep the transmission working properly and extend its lifespan.

[0087] Specifically, for each set of data, after obtaining a global description of the target image contained in the current data, keywords can be extracted from the global description.

[0088] Optionally, the natural language sub-model M in the multimodal model M can be utilized. L From a global perspective, T gExtract the keyword set K and the prompt word P k It can be "from" <quote>< / quote> Extracting latent keywords from the marked content <quote>< / quote> The content within the marker is the global description T. g The above process can be represented by the following mathematical expression: K = M L (T g ,P k )

[0089] Optionally, the Grounded-SAM model M can be used. sam The target image is detected by using each keyword in the keyword set K to obtain the target detection box. The keywords and the target detection boxes are in one-to-one correspondence, and the target detection box set B is obtained.

[0090] Optionally, after obtaining the target detection box set B, the target image is segmented using the target detection boxes in set B to obtain local target images. There is a one-to-one correspondence between the target detection boxes and the local target images. The set of all local target images can be used as the local image set I. loc .

[0091] Optionally, a multimodal model M can be used for the local image set I. loc Each local image in the dataset is processed to generate a corresponding local description, resulting in a local description set T. loc The global description, local image set, local description set, and current data constitute a text-image matching pair. The above describes the process of processing a set of data in the text-image knowledge dataset D. The same method can be used to process all data to obtain the text-image matching pair corresponding to each set of data.

[0092] After the above data augmentation, a new graph and text knowledge dataset D can be obtained. * Graph and text knowledge dataset D * It consists of multiple image-text matching pairs, where the i-th image-text matching pair includes the image I contained in the i-th data set. i Local Image Set I i loc The query question Q contains the i-th set of data. i The i-th data set contains the knowledge text T. i Global description T i g and the local description set T i loc .

[0093] Step 106: Based on the image-text matching pairs corresponding to each set of data, construct positive and negative sample pairs; positive and negative sample pairs are used to train the multimodal retrieval enhancement generation multimodal model.

[0094] Optionally, for the graph-text knowledge dataset D * Each image-text matching pair can be split into positive and negative sample groups; based on the positive and negative sample groups corresponding to each image-text matching pair, positive and negative sample pairs can be constructed; based on the positive and negative sample pairs, the multimodal model to be trained can be compared and learned to obtain a trained multimodal model for actual inference.

[0095] Optionally, the multimodal model can be a multimodal retrieval-augmented generation (RAG) model. The method provided in this application embodiment can improve the generation performance of the multimodal RAG model. A multimodal RAG model is an artificial intelligence model that combines retrieval and generation capabilities. It is mainly used to process and generate data involving multiple modalities (such as text, images, and sound). The following are some typical use cases of multimodal RAG models: 1) Image-to-text generation: Multimodal RAG models can generate descriptive text based on input images, such as automatically generating image titles, descriptions, or stories. 2) Visual question answering systems: In visual question answering systems, multimodal RAG models can process user questions about image content and generate accurate answers. 3) Semantic retrieval of images and text: Multimodal RAG models can retrieve semantically related images from an image database based on text queries provided by users, or retrieve relevant text information based on image queries. The above positive and negative samples can be used to compare and learn the multimodal model to be trained, taking into account the use cases of the multimodal model.

[0096] In the above embodiments, a text-image knowledge dataset is obtained, which includes multiple sets of data. Each set of data includes an image, the corresponding knowledge text, and the corresponding query question. For each set of data, a global description of the target image contained in the current data is obtained. Target detection is performed on the target image based on the keywords contained in the global description to obtain a target detection box. A local image set is obtained based on the target detection box. A local description of each local image in the local image set is obtained to obtain a local description set. The global description, the local image set, the local description set, and the current data constitute a text-image matching pair. Positive and negative sample pairs are constructed based on the text-image matching pairs corresponding to each set of data. The positive and negative sample pairs are used to train the multimodal model. Image detail information is extracted using a fine-grained description approach. Data augmentation is performed on the original text-image knowledge from multiple angles and from global to local perspectives. The multimodal model trained based on the augmented text-image knowledge dataset can output more accurate responses.

[0097] In some embodiments, obtaining a local image set based on a target detection box includes: performing segmentation processing on a target image based on the target detection box to obtain a target local image; obtaining the number of horizontal and vertical segments based on a pre-set image segmentation rule; segmenting the target image into at least one region local image based on the number of horizontal and vertical segments; and constructing a local image set based on the target local image and the region local image.

[0098] Optionally, after obtaining the target detection box set B, the target image is segmented using the target detection boxes in set B to obtain local target images. Since there is a one-to-one correspondence between the target detection boxes and the local target images, the set of all local target images can be used as the target local image set. Since some potential targets may not have been detected during the target detection process described above, this application embodiment supplements these targets.

[0099] Optionally, the pre-set image segmentation rule can be h g *w g Grid, where h g This refers to dividing the image into h equal parts along the y-axis. g Part, i.e., h g For the number of vertical divisions, similarly, w g This refers to dividing the image into w equal parts along the x-axis. g portion, i.e., w g The horizontal segmentation number is used to divide the target image into h segments based on the horizontal and vertical segmentation numbers. g *w g These local images of a region constitute a set of local images of that region. As shown in Figure 4, h g For 3, w g The value is 3, so this image can capture local images of 9 regions.

[0100] Optionally, after obtaining the target local image set and regional local image sets Afterwards, a local image set can be constructed.

[0101] In the above embodiments, the target image is segmented based on the target detection bounding box to obtain a local target image; the number of horizontal and vertical segments is obtained based on a pre-set image segmentation rule; based on the number of horizontal and vertical segments, the target image is segmented into at least one regional local image; and a local image set is constructed based on the target local image and the regional local images. This process supplements targets that may be missed during the target detection process, improving the data augmentation effect.

[0102] In some embodiments, constructing positive and negative sample pairs based on the image-text matching pairs corresponding to each set of data includes: for each image-text matching pair corresponding to each set of data, obtaining the target knowledge text and the target local description set contained in the current image-text matching pair; encoding the target knowledge text to obtain a knowledge text vector; encoding each local description in the target local description set to obtain a local text vector for each local description; splitting the current image-text matching into positive and negative sample groups based on the knowledge text vector and the local text vector of each local description; and constructing positive and negative sample pairs based on the positive and negative sample groups obtained from splitting the image-text matching pairs corresponding to each set of data.

[0103] Optionally, since irrelevant descriptions may exist in the local description set, such as the local description corresponding to a local image of a region, the likelihood of these descriptions being irrelevant is high. Therefore, for each image-text matching pair corresponding to a set of data, the target knowledge text and the target local description set contained in the current image-text matching pair can be obtained; the text embedding model M of the multimodal model M is then used. te Encode the target knowledge text into a knowledge text vector V. i T Encode each local description in the target local description set into a local text vector. Based on the knowledge text vector and the local text vector of each local description, the current image-text match is split into positive and negative sample groups; based on the positive and negative sample groups obtained from the split image-text match pairs corresponding to each group of data, positive and negative sample pairs are constructed.

[0104] The above embodiments provide a method for constructing positive and negative sample pairs, which can then be used to perform comparative learning on the multimodal model to be trained. Since the positive and negative sample pairs are constructed based on the data-augmented graph and text knowledge dataset, the trained multimodal model can output more accurate responses.

[0105] In some embodiments, based on the knowledge text vector and the local text vector of each local description, the current image-text matching is split into positive and negative sample groups, including: for each local description, calculating the cosine distance between the knowledge text vector and the local text vector of the current local description; if the cosine distance is greater than or equal to a preset cosine distance threshold, the current local description is taken as a positive sample of the local description; if the cosine distance is less than the preset cosine distance threshold, the current local description is taken as a negative sample of the local description; all positive samples of local descriptions constitute a positive sample set of local descriptions, and all negative samples of local descriptions constitute a negative sample set of local descriptions; based on the positive sample set of local descriptions, the local image corresponding to each local description in the positive sample set of local descriptions, the corresponding global description, and the corresponding data, a positive sample group is constructed; based on the negative sample set of local descriptions and the local image corresponding to each local description in the negative sample set of local descriptions, a negative sample group is constructed.

[0106] Optionally, a text embedding model M using a multimodal model M. te Encode the target knowledge text into a knowledge text vector V. i T Encode each local description in the target local description set into a local text vector. Then, calculate V. i T and The cosine distance between them is used to classify local descriptions that are greater than or equal to a preset cosine distance threshold as positive local description samples; all positive local description samples constitute the positive local description sample set T. i loc_p Otherwise, they are classified as local description negative samples; all local description negative samples constitute the local description negative sample set T. i loc_n It can obtain a locally descriptive positive sample set T. i loc_p Corresponding local image positive sample set Obtain the local description of the negative sample set T i loc_n Corresponding local image negative sample set I i loc_n Therefore, the i-th image-text matching pair can form positive and negative sample groups, with the positive sample group being... The negative sample group is According to this allocation mechanism, dataset D * Further optimization into a graph-text knowledge dataset D ** .

[0107] The above embodiments provide an implementation method for splitting image-text matching into positive and negative sample groups. This process is the foundation for constructing positive and negative sample pairs, which can then be used to perform comparative learning on the multimodal model to be trained. Since the positive and negative sample pairs are constructed based on the data-augmented image-text knowledge dataset, the trained multimodal model can output more accurate responses.

[0108] In some embodiments, constructing positive and negative sample pairs based on the positive and negative sample groups obtained by splitting the image-text matching pairs corresponding to each set of data includes: for each set of data corresponding to the image-text matching pairs, obtaining the target positive and negative sample groups obtained by splitting the current image-text matching pairs, and taking the positive sample groups in the target positive and negative sample groups as a positive sample pair; randomly selecting target data from all data in the image-text knowledge dataset except for the data corresponding to the current image-text matching pairs, and constructing negative sample pairs based on the negative sample groups in the target positive and negative sample groups, the global description of the image contained in the target data, and the target data.

[0109] Optionally, for the graph-text knowledge dataset D ** The positive sample group obtained by splitting the i-th image-text matching pair Images and text within the text can form positive sample pairs, while negative sample pairs... Other data, including global descriptions of the images contained in other data, form negative sample pairs with each other.

[0110] The above embodiments provide an implementation method for constructing positive and negative sample pairs after splitting each image-text matching pair into positive and negative sample groups. Subsequently, based on the positive and negative sample pairs, the multimodal model to be trained can be compared and learned to output more accurate responses.

[0111] In some embodiments, the data processing method provided in this application further includes: performing comparative learning on the multimodal model to be trained based on positive and negative sample pairs to obtain a basic multimodal model; generating an instruction fine-tuning dataset based on the image-text matching pairs corresponding to each set of data and a preset instruction template; and performing instruction fine-tuning training on the basic multimodal model based on the instruction fine-tuning dataset to obtain a trained multimodal model.

[0112] Contrastive learning is a self-supervised learning technique that learns feature representations by comparing positive and negative sample pairs. Its core principle is to bring positive sample pairs closer together and negative sample pairs further apart.

[0113] Optionally, LoRA is a parameter-efficient fine-tuning (PEFT) technique. Applying LoRA not only allows for efficient model training but also avoids underfitting caused by insufficient data for full model training. This application's embodiments may use LoRA for parameter fine-tuning.

[0114] Among them, instruction fine-tuning technology enables the model to learn to follow instructions to complete specific tasks, improving the model's responsiveness to specific instructions or task requests. In this embodiment, based on the basic multimodal model obtained through comparative learning, instruction fine-tuning training can be further performed to improve retrieval efficiency and effectiveness in practical application scenarios. Because real-time response considerations prevent local image extraction as described above during the retrieval process, the fine-tuned model needs to have the ability to extract local image feature information through instructions and keywords. To achieve this, this embodiment pre-sets an instruction template and uses this template to train the dataset D. ** Transformed into an instruction fine-tuning dataset for instruction fine-tuning

[0115] For example, a preset instruction template can be:

[0116] ###Summarize

[0117] <Global Description>

[0118] ###Knowledge

[0119] <Knowledge Text>

[0120] ###Keywords

[0121] <Knowledge Text Keyword 1>

[0122] <Knowledge Text Keyword 2>

[0123] ...

[0124] <Knowledge Text Keywords n>

[0125] ###question

[0126] <Query Question>

[0127] ###instruction

[0128] <Image Local Description Preset Command>

[0129] ###detail

[0130] <Partial Description 1>

[0131] <Partial Description 2>

[0132] ...

[0133] <Local description m>

[0134] Among them, the <Image Local Description Preset Instructions> requires a pre-set instruction set; other content can be obtained from dataset D. ** Retrieved from [the image]. <Image Local Description Preset Commands> can be found in the following commands (including but not limited to):

[0135] 1. Based on the question and your knowledge, please help me extract the main key content from the image.

[0136] 2. Based on the question, use your knowledge to analyze the content of the image and find the key information points.

[0137] 3. Based on the question, identify the key elements in the image.

[0138] 4. Generate a concise description of the image content, highlighting key information related to the question and knowledge.

[0139] 5. Based on the image content, knowledge, and question, recommend relevant key search terms for me.

[0140] 6. Determine the theme or focus of the image and summarize it using keywords.

[0141] Optionally, fine-tune the dataset after receiving instructions. Subsequently, LoRA can be used to fine-tune the basic multimodal model using instructions, resulting in a trained multimodal model M. * .

[0142] The above embodiments describe the training process of the multimodal model. Fine-grained description and fine-tuning enable the multimodal model to have the ability to extract local key information and align local multimodal features, thereby improving the multimodal model's ability to extract image and text embedding vectors of knowledge in a certain domain and enhancing the fine-grainedness of image and text embedding vectors.

[0143] In some embodiments, a data processing method is provided, as shown in Figure 5, comprising the following steps: For each set of data in the image-text knowledge dataset, the data includes an image, knowledge text corresponding to the image, and a query question corresponding to the knowledge text; therefore, this data is also called multimodal data. First, a global description of the image content is generated. Then, keyword extraction based on the global description and image target detection based on the keywords are performed. Then, target image segmentation and image preset region segmentation are performed. Then, a description of the local image content is generated. Then, comparative learning is performed based on the combination of fine-grained image-text positive and negative samples. Then, the template is fine-tuned based on the image local description instructions, and instruction fine-tuning is performed. Finally, a fine-tuned multimodal model is output.

[0144] In some embodiments, the data processing method provided in this application further includes: acquiring data to be imported, the data to be imported including an image to be imported, imported knowledge text corresponding to the image to be imported, and imported query questions corresponding to the imported knowledge text; acquiring a global description of the image to be imported as the global description of imported data, performing target detection on the image to be imported based on the keywords contained in the global description of imported data, and acquiring a set of local images to be imported based on the detection results; acquiring a local description of each local image in the set of local images to be imported, and obtaining a set of local descriptions to be imported; encoding the global description of imported data to obtain a global description vector to be imported; and encoding the imported knowledge text to obtain the imported data. Knowledge text vectors are generated by encoding each local description in the local description set and obtaining the local text vector. The cosine distance between the knowledge text vector and each local text vector is calculated, and the local descriptions in the local description set are sorted in ascending order of cosine distance to obtain the top-ranked target local descriptions. The local image embedding vector of the local image corresponding to each target local description is extracted. The global image embedding vector of the image is extracted. Based on the global description vector, target local descriptions, local image embedding vectors, global image embedding vectors, and knowledge text vectors, knowledge groups are constructed and stored in the knowledge base.

[0145] Optionally, the steps of obtaining a global description of the imported images, performing object detection on the imported images based on keywords contained in the global description, obtaining a local image set based on the detection results, and obtaining a local description of each local image in the local image set can be performed using the Grounded-SAM model and the multimodal model M. * The detailed implementation process can be found in the aforementioned embodiments and will not be repeated here.

[0146] Among them, the multimodal model M is used. * Text embedding model The global description of the database is encoded to obtain a global description vector, and the knowledge text is encoded to obtain a knowledge text vector. Each local description in the local description set is then encoded to obtain a local text vector. The cosine distance between the knowledge text vector and each local text vector is calculated, and the local descriptions in the local description set are sorted in ascending order of cosine distance. The local description with the closest TOP-K distance (K value set according to different situations) is selected as the target local description. A multimodal model M is then used. * Image embedding model Extract the local image embedding vector of the local image corresponding to each target local description; extract the global image embedding vector of the image to be imported. Combine the global description vector, target local description, local image embedding vector, global image embedding vector, imported knowledge text vector, data to be imported, imported global description, and the local image corresponding to the target local description into a knowledge group and store it in the knowledge base.

[0147] In some embodiments, referring to Figure 6, the data entry step includes: for the multimodal data to be entered into the database, firstly, a global description of the image content can be generated; then, keyword extraction based on the global description and image target detection based on the keywords can be performed; then, target image segmentation and image preset region segmentation can be performed; then, a description of the local image content can be generated; then, embedding vectors are extracted from all texts, and the cosine distance between the knowledge text embedding vector and all generated text embedding vectors is calculated; the TOP-K nearest neighbors are used to generate text and corresponding local images, embedding vectors are extracted from the original image and the selected local images, and the effective embedding vectors are entered into the database.

[0148] In the above embodiments, a multimodal fine-grained embedding vector matching mechanism is proposed to optimize the current mainstream data entry strategy.

[0149] Regarding retrieval strategies, the current mainstream approach is to first expand the query question, and then search the knowledge base based on the original query question and multiple expanded query questions. While expanding the query question can improve the hit rate, if the knowledge base is large and the query question is not clearly stated, it can lead to unsatisfactory retrieval results and inaccurate generated results. Therefore, this invention adopts a method of "global coarse matching of image description and fine matching of local description and query question" to optimize the current retrieval strategy.

[0150] In some embodiments, the data processing method provided in this application further includes: receiving a retrieval request, the retrieval request including a query image and a query question; obtaining a global description of the query image; retrieving at least one target knowledge group in a knowledge base whose similarity meets preset conditions based on the global description of the query image; randomly selecting an image local description instruction from a preset set of image local description instructions, and generating multiple local descriptions based on the image local description instruction, the global description, the query question, and the knowledge text in the target knowledge group; expanding the query question based on the multiple local descriptions to obtain an expanded question; retrieving the final knowledge group in at least one target knowledge group based on the query question and the expanded question, and generating a retrieval response based on the final knowledge group.

[0151] Among them, it can be achieved through the multimodal model M * The query image is processed to generate a global description of it. A multimodal model M is used. * Text embedding model This global description is processed to generate a query text vector.

[0152] Specifically, at least one target knowledge group that meets preset similarity criteria can be retrieved from the knowledge base, with a maximum similarity of N (the value of N may vary depending on the specific situation). Specifically, the global description vector of each knowledge group in the knowledge base can be obtained, the cosine distance between the query text vector and the global description vector can be calculated, and the knowledge groups in the knowledge base can be sorted in ascending order of cosine distance. The top N knowledge groups are then selected as the target knowledge groups.

[0153] As described above, the preset instruction set for image local description includes, but is not limited to, the following instructions: 1. Based on the question and your knowledge, please help me extract the main key content from the image. 2. Based on the question, use your knowledge to analyze the image content and find the key information points. 3. Based on the question, find the key elements in the image. 4. Generate a concise description of the image content, highlighting the key content related to the question and knowledge. 5. Based on the image content, knowledge, and question, recommend relevant key search descriptions. 6. Determine the theme or focus of the image and summarize it using keywords.

[0154] Among them, a pre-trained multimodal model M can be used based on randomly selected local image description instructions, global descriptions of the queried image, query questions, and knowledge text in the target knowledge group. * Generate multiple local descriptions. Based on these local descriptions, extensible instructions can be used to enable multimodal models M... * The query question is expanded in a targeted manner. Finally, the query question and expanded questions are used to retrieve information from at least one target knowledge group to obtain the final knowledge group. The knowledge text in the final knowledge group is then converted into knowledge bars, and a search response is generated based on the knowledge bars. Since the generation of local descriptions no longer requires the preprocessing of local images, the timeliness of the search is guaranteed, while the accuracy and granularity of the search are also improved, ultimately enhancing the quality of the search response generation.

[0155] In some embodiments, referring to Figure 7, the retrieval steps include: receiving a retrieval request, which includes a query image and a query question; performing a global description generation of the image content; extracting the embedding vector of the global description and detecting the top-N knowledge groups from the knowledge base; then generating a local description through instructions; expanding the query question; extracting the embedding vector of the expanded question; retrieving knowledge entries within the top-N knowledge groups and generating an answer by combining the retrieved knowledge entries.

[0156] In the above embodiments, the image is first described globally using a multimodal model, then the top-N similar knowledge groups are retrieved from the knowledge base, and finally, relevant answers are located in the knowledge groups through query questions and extended questions, thereby improving the retrieval response generation effect.

[0157] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0158] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.

[0159] In one exemplary embodiment, a data processing apparatus is provided, comprising:

[0160] The acquisition module is configured to acquire a graph and text knowledge dataset, which includes multiple sets of data. Each set of data includes an image, the knowledge text corresponding to the image, and the query question corresponding to the knowledge text.

[0161] The construction module is configured to, for each set of data, obtain a global description of the target image contained in the current data, perform target detection on the target image based on the keywords contained in the global description to obtain a target detection box, obtain a local image set based on the target detection box, obtain a local description set for each local image in the local image set, and then obtain a local description set. The global description, the local image set, the local description set, and the current data constitute an image-text matching pair.

[0162] The training module is configured to construct positive and negative sample pairs based on the image-text matching pairs corresponding to each set of data; and to train the multimodal model based on the positive and negative sample pairs.

[0163] In some embodiments, the acquisition module is specifically configured as follows:

[0164] Based on the target detection bounding box, the target image is segmented to obtain a local image of the target;

[0165] Based on the pre-set image segmentation rules, the number of horizontal and vertical segments is obtained, and the target image is segmented into at least one local image region based on the number of horizontal and vertical segments.

[0166] A local image set is constructed based on the target local image and the region local image.

[0167] In some embodiments, the training module is specifically configured as follows:

[0168] For each set of data corresponding to the image-text matching pair, obtain the target knowledge text and target local description set contained in the current image-text matching pair; encode the target knowledge text to obtain the knowledge text vector; encode each local description in the target local description set to obtain the local text vector of each local description;

[0169] Based on the knowledge text vector and the local text vector of each local description, the current image-text match is split into positive and negative sample groups;

[0170] Positive and negative sample pairs are constructed based on the positive and negative sample groups obtained by splitting the image and text matching pairs corresponding to each set of data.

[0171] In some embodiments, the training module is specifically configured as follows:

[0172] For each local description, calculate the cosine distance between the knowledge text vector and the local text vector of the current local description. If the cosine distance is greater than or equal to a preset cosine distance threshold, the current local description is taken as a positive sample of the local description; if the cosine distance is less than the preset cosine distance threshold, the current local description is taken as a negative sample of the local description. All positive samples of local descriptions constitute the positive sample set of local descriptions, and all negative samples of local descriptions constitute the negative sample set of local descriptions.

[0173] A positive sample group is constructed based on the local description positive sample set, the local image corresponding to each local description in the local description positive sample set, the corresponding global description, and the corresponding data.

[0174] A negative sample group is constructed based on the local description negative sample set and the local image corresponding to each local description in the local description negative sample set.

[0175] In some embodiments, the training module is specifically configured as follows:

[0176] For each set of data corresponding to the image and text matching pair, obtain the target positive and negative sample groups obtained by splitting the current image and text matching pair, and take the positive sample group in the target positive and negative sample group as a positive sample pair;

[0177] Randomly select target data from all data in the image-text knowledge dataset except for the data corresponding to the current image-text matching pair. Construct negative sample pairs based on the negative sample group in the target positive and negative sample group, the global description of the image contained in the target data, and the target data.

[0178] In some embodiments, the data processing apparatus further includes an ingestion module, configured to:

[0179] Obtain the data to be imported, which includes the images to be imported, the corresponding knowledge text for the images to be imported, and the corresponding query questions for the knowledge text to be imported.

[0180] Obtain a global description of the imported images as the global description of the imported images. Perform target detection on the imported images based on the keywords contained in the global description of the imported images. Obtain a set of local images in the imported images based on the detection results. Obtain a local description of each local image in the set of local images in the imported images to obtain a set of local descriptions in the imported images.

[0181] The global description of the database is encoded to obtain the global description vector; the knowledge text of the database is encoded to obtain the knowledge text vector; and each local description in the local description set of the database is encoded to obtain the local text vector.

[0182] Calculate the cosine distance between the knowledge text vector and each local text vector in the database, and sort the local descriptions in the local description set in the database according to the order of cosine distance from smallest to largest, and obtain the target local descriptions at the top of the list.

[0183] Extract the local image embedding vector of the local image corresponding to the local description of each target; extract the global image embedding vector of the images in the database;

[0184] Based on the global description vector, target local description, local image embedding vector, global image embedding vector, and knowledge text vector, knowledge groups are constructed and stored in the knowledge base.

[0185] In some embodiments, the data processing apparatus further includes a retrieval module configured to:

[0186] Receive search requests, which include query images and query questions;

[0187] Obtain a global description of the queried image;

[0188] Based on the global description of the query image, at least one target knowledge group whose similarity meets the preset conditions is retrieved from the knowledge base;

[0189] Randomly select one image local description instruction from the preset instruction set for image local description, and generate multiple local descriptions based on the image local description instruction, global description, query question, and knowledge text in the target knowledge group;

[0190] The extended query problem is obtained by expanding the query problem based on multiple local descriptions;

[0191] Based on the query question and extended questions, a search is conducted in at least one target knowledge group to obtain the final knowledge group, and a search response is generated based on the final knowledge group.

[0192] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0193] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram is shown in Figure 8. The computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device stores a dataset of graphic and textual knowledge. The I / O interfaces of the computer device are used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing method.

[0194] Those skilled in the art will understand that the structure shown in Figure 8 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or may combine certain components, or may have different component arrangements.

[0195] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0196] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0197] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0198] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0199] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0200] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: Obtain a text-image knowledge dataset, which includes multiple sets of data, each set of data including an image, knowledge text corresponding to the image, and a query question corresponding to the knowledge text; For each set of data, obtain a global description of the target image contained in the current data; Target detection is performed on the target image based on the keywords contained in the global description to obtain a target detection box; A local image set is obtained based on the target detection box; A local description is obtained for each local image in the local image set to form a local description set; the global description, the local image set, the local description set, and the current data constitute an image-text matching pair; Based on the image-text matching pairs corresponding to each set of data, positive and negative sample pairs are constructed; and The multimodal model is trained based on the positive and negative sample pairs.

2. The method according to claim 1, characterized in that, The step of obtaining a local image set based on the target detection box includes: Based on the target detection box, the target image is segmented to obtain a local image of the target; The number of horizontal and vertical segments is obtained based on pre-set image segmentation rules; Based on the number of horizontal segments and the number of vertical segments, the target image is divided into at least one regional local image; and A local image set is constructed based on the target local image and the region local image.

3. The method according to claim 1, characterized in that, The construction of positive and negative sample pairs based on the image-text matching pairs corresponding to each set of data includes: For each set of data corresponding to the image-text matching pair, obtain the target knowledge text and target local description set contained in the current image-text matching pair; The target knowledge text is encoded to obtain a knowledge text vector; Each local description in the target local description set is encoded to obtain a local text vector for each local description. Based on the knowledge text vector and the local text vector of each local description, the current image-text matching is split into positive and negative sample groups; and Positive and negative sample pairs are constructed based on the positive and negative sample groups obtained by splitting the image and text matching pairs corresponding to each set of data.

4. The method according to claim 3, characterized in that, The step of splitting the current image-text matching into positive and negative sample groups based on the knowledge text vector and the local text vector of each local description includes: For each local description, calculate the cosine distance between the knowledge text vector and the local text vector of the current local description; If the cosine distance is greater than or equal to a preset cosine distance threshold, the current local description is taken as a positive sample of the local description. If the cosine distance is less than the preset cosine distance threshold, the current local description is taken as a negative local description sample; all positive local description samples constitute a positive local description sample set, and all negative local description samples constitute a negative local description sample set. Based on the local description positive sample set, the local image corresponding to each local description in the local description positive sample set, the corresponding global description, and the corresponding data, a positive sample group is constructed; and Based on the local description negative sample set and the local image corresponding to each local description in the local description negative sample set, a negative sample group is constructed.

5. The method according to claim 3, characterized in that, The construction of positive and negative sample pairs based on the positive and negative sample groups obtained by splitting the image and text matching pairs corresponding to each group of data includes: For each set of data corresponding to the image-text matching pair, obtain the target positive and negative sample groups obtained by splitting the current image-text matching pair; The positive sample group in the target positive and negative sample group is taken as a positive sample pair; Randomly select target data from all data in the image-text knowledge dataset, excluding the data corresponding to the current image-text matching pair; and Based on the negative sample group in the target positive and negative sample group, the global description of the image contained in the target data, and the target data, a negative sample pair is constructed.

6. The method according to claim 1, characterized in that, The training of the multimodal model based on the positive and negative sample pairs includes: Based on the positive and negative sample pairs, comparative learning is performed on the multimodal model to be trained to obtain the basic multimodal model; Based on the image-text matching pairs and preset instruction templates corresponding to each set of data, an instruction fine-tuning dataset is generated; and Based on the instruction fine-tuning dataset, the basic multimodal model is trained by instruction fine-tuning to obtain the trained multimodal model.

7. The method according to claim 1, characterized in that, The method further includes: Obtain the data to be entered into the database, which includes the image to be entered into the database, the knowledge text corresponding to the image to be entered into the database, and the query question to be entered into the database corresponding to the knowledge text to be entered into the database. Obtain a global description of the image to be imported into the database, and use it as the global description for importing into the database; Target detection is performed on the imported images based on the keywords contained in the global description of the imported images; A local image set is obtained for storage based on the detection results; Obtain the local description of each local image in the local image set to obtain the local description set. The global description of the database entry is encoded to obtain the global description vector of the database entry; The knowledge text to be stored is encoded to obtain a knowledge text vector. Each local description in the local description set is encoded to obtain a local text vector for the database. Calculate the cosine distance between the imported knowledge text vector and each imported local text vector; The local descriptions in the local description set are sorted in ascending order of cosine distance; Retrieve the top-ranked, preset number of target local descriptions; Extract the local image embedding vector corresponding to the local image of each target local description; Extract the global image embedding vector of the imported image; and Based on the global description vector, the target local description, the local image embedding vector, the global image embedding vector, and the knowledge text vector, a knowledge group is constructed and stored in the knowledge base.

8. The method according to claim 1, characterized in that, The method further includes: Receive a search request, which includes a query image and a query question; Obtain a global description of the queried image; Based on the global description of the query image, at least one target knowledge group whose similarity meets the preset conditions is retrieved from the knowledge base; Randomly select one image local description instruction from the preset instruction set for image local description; Based on the image local description instructions, the global description, the query question, and the knowledge text in the target knowledge group, multiple local descriptions are generated using a trained multimodal model; Based on the aforementioned multiple local descriptions, the query problem is expanded, resulting in an extended problem. Based on the query question and the extended question, a final knowledge group is obtained by retrieving information from at least one target knowledge group; and A search response is generated based on the final knowledge group.

9. A data processing apparatus, characterized in that, The device includes: The acquisition module is configured to acquire a graphic knowledge dataset, which includes multiple sets of data, each set of data including an image, knowledge text corresponding to the image, and a query question corresponding to the knowledge text; The construction module is configured to, for each set of data, obtain a global description of the target image contained in the current data, perform target detection on the target image based on the keywords contained in the global description to obtain a target detection box, obtain a local image set based on the target detection box, obtain a local description set for each local image in the local image set, and obtain a local description set; the global description, the local image set, the local description set, and the current data constitute an image-text matching pair; The training module is configured to construct positive and negative sample pairs based on the image-text matching pairs corresponding to each set of data; and to train the multimodal model based on the positive and negative sample pairs.

10. The apparatus according to claim 9, characterized in that, The acquisition module is specifically configured as follows: Based on the target detection box, the target image is segmented to obtain a local image of the target; The number of horizontal and vertical segments is obtained based on pre-set image segmentation rules; Based on the number of horizontal segments and the number of vertical segments, the target image is divided into at least one regional local image; and A local image set is constructed based on the target local image and the region local image.

11. The apparatus according to claim 9, characterized in that, The training module is specifically configured as follows: For each set of data corresponding to the image-text matching pair, obtain the target knowledge text and target local description set contained in the current image-text matching pair; The target knowledge text is encoded to obtain a knowledge text vector; Each local description in the target local description set is encoded to obtain a local text vector for each local description. Based on the knowledge text vector and the local text vector of each local description, the current image-text matching is split into positive and negative sample groups; and Positive and negative sample pairs are constructed based on the positive and negative sample groups obtained by splitting the image and text matching pairs corresponding to each set of data.

12. The apparatus according to claim 11, characterized in that, The training module is specifically configured as follows: For each local description, calculate the cosine distance between the knowledge text vector and the local text vector of the current local description; If the cosine distance is greater than or equal to a preset cosine distance threshold, the current local description is taken as a positive sample of the local description. If the cosine distance is less than the preset cosine distance threshold, the current local description is taken as a negative local description sample; all positive local description samples constitute a positive local description sample set, and all negative local description samples constitute a negative local description sample set. Based on the local description positive sample set, the local image corresponding to each local description in the local description positive sample set, the corresponding global description, and the corresponding data, a positive sample group is constructed; and Based on the local description negative sample set and the local image corresponding to each local description in the local description negative sample set, a negative sample group is constructed.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images