Multimodal question-answering method and related device

Abstract

The present application relates to the field of AI, and in particular to a multimodal question-answering method and a related device. The method comprises: encoding each frame of a target video so as to obtain a first encoding result for each frame; a question-answering device determining at least two of a similarity score, a matching score, and a discriminability score for each frame on the basis of the first encoding result for each frame of the target video and target question text; on the basis of at least two of the similarity score, the matching score, and the discriminability score for each frame, the question-answering device selecting K first encoding results from among the first encoding results corresponding to all frames of the target video, wherein K is an integer greater than 1; and the question-answering device obtaining an answer to a target question on the basis of the K first encoding results and the target question text. The use of the solution of the present application can improve the accuracy of a question-answering result.

Description

Multimodal question answering methods and related equipment

[0001] This application claims priority to Chinese Patent Application No. 202411554501.4, filed with the China National Intellectual Property Administration on October 31, 2024, entitled "Multimodal Question Answering Method and Related Device", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI), and more particularly to a multimodal question answering method and related devices. Background Technology

[0003] Multimodal dialogue systems are an emerging artificial intelligence technology that can interact with users through various input methods (such as voice, text, and images) and provide information and services through various output methods (such as voice, text, and images). Video question answering is based on input video and question text to provide answers. One specific approach is to select multiple frames from the input video and then provide an answer based on the selected frames and the input question text. Methods for selecting multiple frames from the input video include uniform frame selection or random frame selection. However, this frame selection method is inefficient, and the selected frames may introduce noise into the answer, leading to low accuracy. Summary of the Invention

[0004] This application provides a multimodal question-answering method and related equipment, which can improve the accuracy of the answer results.

[0005] Firstly, embodiments of this application provide a multimodal question-answering method. The method includes:

[0006] The question-answering device acquires a target video and a target question text; the device encodes each frame in the target video to obtain a first encoding result for each frame; based on the first encoding result of each frame in the target video and the target question text, the device determines at least two of the following for each frame: a similarity score, a matching score, and a discrimination score. The similarity score characterizes the degree of similarity between each frame and the target question, the matching score characterizes the degree of matching between each frame and the target question, and the discrimination score characterizes the degree of discrimination between each frame and other video frames in the target video. Based on at least two of the similarity score, matching score, and discrimination score for each frame, the device selects K first encoding results from the first encoding results corresponding to all frames in the target video, where K is an integer greater than 1. Based on the K first encoding results and the target question text, the device obtains the answer to the target question.

[0007] It can be seen that when selecting video frames for video question answering, selecting the first encoding result strongly related to the target question from the first encoding results corresponding to all video frames in the target video based on at least one of the matching score and similarity score between each frame and the target question text, and then using this first encoding result for subsequent processing, helps improve the accuracy of the answer. Furthermore, considering the discriminative information of video frames during video question answering ensures that the selected first encoding result contains more information, and subsequent processing based on this first encoding result further improves the accuracy of the answer. Simultaneously, selecting K first encoding results from the first encoding results corresponding to all frames in the target video based on at least two of the similarity score, matching score, and discriminative score for each frame improves frame selection efficiency, thereby improving question answering efficiency.

[0008] In conjunction with the first aspect, in a feasible implementation, the question-answering device selects K first-encoded results from the first-encoded results corresponding to all frames in the target video based on at least two of the similarity score, matching score, and discriminative score for each frame, including:

[0009] The question-answering device calculates the overall score of each frame based on at least two of the similarity score, matching score, and discrimination score. Based on the overall score of each frame, the question-answering device sorts the first encoding results of the video frames in the target video in descending order, and selects the top K first encoding results from the sorted results.

[0010] It can be seen that selecting K first encoding results from the first encoding results corresponding to all frames in the target video based on at least two of the similarity score, matching score and discrimination score of each frame is beneficial to improving frame selection efficiency, and thus beneficial to improving question answering efficiency.

[0011] In conjunction with the first aspect, in a feasible implementation, the question-answering device selects K first-encoded results from the first-encoded results corresponding to all frames in the target video based on at least two of the similarity score, matching score, and discriminative score for each frame, including:

[0012] The question-answering device calculates the overall score of each frame based on at least two of the similarity score, matching score, and discrimination score; the question-answering device determines the selection probability of each frame based on the overall score of each frame, and the selection probability of each frame is the probability that the first encoded result of each frame is selected from the first encoded results corresponding to all frames in the target video; the question-answering device selects K first encoded results from the first encoded results corresponding to all frames in the target video based on the selection probability of each frame.

[0013] By selecting frames probabilistically, the first encoding result corresponding to all frames in the target video may be selected, which helps to improve the stability of the accuracy of the response results obtained by adopting the solution of this application.

[0014] In conjunction with the first aspect, in a feasible implementation, the question-answering device determines a similarity score, a matching score, and a discriminative score for each frame based on the first encoding result of each frame in the target video and the target question text, including:

[0015] The question-answering device extracts features from the first encoding result of each frame to obtain the first feature vector of each frame; it encodes the target question text to obtain the second encoding result; the question-answering device calculates the similarity score of each frame based on the first feature vector and the second encoding result of each frame; the question-answering device performs multimodal fusion encoding on the first encoding result and the target question text of each frame to obtain the third encoding result corresponding to each frame; it calculates the matching score of each frame based on the third encoding result corresponding to each frame; the question-answering device normalizes the first encoding result of each frame to obtain the normalized result of each frame; and the question-answering device calculates the discriminative score of each frame in the target video based on the normalized result of all video frames in the target video.

[0016] In conjunction with the first aspect, in a feasible implementation, the question-answering device calculates the discriminative score for each frame in the target video based on the normalized processing results of all video frames in the target video, including:

[0017] The question-answering device calculates the similarity between the i-th frame and each of the other video frames in the target video, based on the normalized result of the i-th frame and the normalized result of the other video frames in the target video. The question-answering device calculates the average similarity between the i-th frame and each of the other video frames in the target video. The question-answering device determines the discrimination score of the i-th frame based on the average value. The discrimination score of each frame in the target video includes the discrimination score of the i-th frame.

[0018] Secondly, embodiments of this application provide a question-and-answer device. The question-and-answer device includes an acquisition unit, an encoding unit, a determination unit, and a selection unit.

[0019] The acquisition unit is used to acquire the target video and the target question text.

[0020] The encoding unit is used to encode each frame in the target video to obtain the first encoding result of each frame;

[0021] The determining unit is used to determine at least two of the following for each frame: similarity score, matching score, and discrimination score, based on the first encoding result of each frame in the target video and the target question text: the similarity score is used to characterize the degree of similarity between each frame and the target question, the matching score is used to characterize the degree of matching between each frame and the target question, and the discrimination score is used to characterize the degree of discrimination between each frame and other video frames in the target video other than each frame.

[0022] A selection unit is used to select K first encoding results from the first encoding results corresponding to all frames in the target video based on at least two of the similarity score, matching score, and discriminative score of each frame; where K is an integer greater than 1.

[0023] The unit is also used to determine the answer to the target question based on the K first encoding results and the target question text.

[0024] In conjunction with the second aspect, in a feasible implementation, the selected unit is specifically used for:

[0025] The overall score of each frame is calculated based on at least two of the similarity score, matching score, and discrimination score. The first encoding results of the video frames in the target video are sorted from high to low based on the overall score of each frame, and the top K first encoding results are selected from the sorted results.

[0026] In conjunction with the second aspect, in a feasible implementation, the selected unit is specifically used for:

[0027] The overall score of each frame is calculated based on at least two of the similarity score, matching score, and discriminative score. The selection probability of each frame is determined based on the overall score of each frame. The selection probability of each frame is the probability that the first encoded result of each frame is selected from the first encoded results corresponding to all frames in the target video. K first encoded results are selected from the first encoded results corresponding to all frames in the target video based on the selection probability of each frame.

[0028] In conjunction with the second aspect, in a feasible implementation, the unit is specifically used for:

[0029] Feature extraction is performed on the first encoding result of each frame to obtain the first feature vector of each frame; the target question text is encoded to obtain the second encoding result; the similarity score of each frame is calculated based on the first feature vector and the second encoding result of each frame; multimodal fusion encoding is performed on the first encoding result and the target question text of each frame to obtain the third encoding result corresponding to each frame; the matching score of each frame is calculated based on the third encoding result corresponding to each frame; the first encoding result of each frame is normalized to obtain the normalized result of each frame; the discriminative score of each frame in the target video is calculated based on the normalized result of all video frames in the target video.

[0030] In conjunction with the second aspect, in a feasible implementation, regarding the calculation of the discrimination score for each frame in the target video based on the normalization processing results of all video frames in the target video, the determining unit is specifically used for:

[0031] The similarity between frame i and each of the target video frames other than frame i is calculated based on the normalized result of frame i and the normalized result of each of the target video frames other than frame i. The average similarity between frame i and each of the target video frames other than frame i is calculated. The discrimination score of frame i is determined based on the average value. The discrimination score of each frame in the target video includes the discrimination score of frame i.

[0032] Thirdly, embodiments of this application provide a question-and-answer device, including a processor and a memory. The memory is used to store program code. The processor is used to invoke the program code stored in the memory to execute the method provided in the first aspect or any possible implementation of the first aspect.

[0033] Fourthly, embodiments of this application provide a computer storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform a method as provided in any possible implementation of the first aspect.

[0034] Fifthly, embodiments of this application provide a computer program product that, when run on a computer, causes the computer to perform the method provided in any possible implementation of the first aspect.

[0035] It is understood that the question-answering device described in the second or third aspect above is used to execute any of the methods provided in the first aspect, and the computer storage medium described in the fourth aspect and the computer program product described in the fifth aspect are both used to implement any of the methods provided in the first aspect. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0036] Figure 1 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0037] Figure 2 is a schematic diagram of the logical architecture of a multimodal question answering method provided in an embodiment of this application;

[0038] Figure 3 is a flowchart illustrating a multimodal question-answering method provided in an embodiment of this application;

[0039] Figure 4 is a schematic diagram of a question-answering model training architecture provided in an embodiment of this application;

[0040] Figure 5 is a structural schematic diagram of a question-and-answer device provided in an embodiment of this application;

[0041] Figure 6 is a schematic diagram of another question-and-answer device provided in an embodiment of this application. Detailed Implementation

[0042] The terms “first,” “second,” “third,” and “fourth,” etc., used in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order.

[0043] "Multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating three possible relationships. For example, A and / or B means: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0044] The embodiments of this application will now be described with reference to the accompanying drawings.

[0045] Referring to Figure 1, Figure 1 is a schematic diagram of a system architecture provided in an embodiment of this application. As shown in Figure 1, the system includes a data collection device 10 (optional), a server 11, and a client 12 (optional). The data collection device 10, the server 11, and the client are connected through a communication network.

[0046] The data collection device 10 is a device that can be used to collect video data, image data and / or text data. For example, the data collection device 10 can be a device with a camera, such as a smartphone, smartwatch, smart bracelet, desktop computer, laptop computer, tablet and other devices.

[0047] Server 11 is a device capable of data processing, such as a cloud server, distributed server, rack server, blade server, tower server, etc.

[0048] Client 12 can be user devices, such as smartphones, smartwatches, smart bracelets, desktop computers, laptops, tablets, and other devices.

[0049] The data collection device 10 acquires training samples through various means (manual input, online searches, etc.). These training samples include question text, videos, and answers. The data collection device 10 transmits the training samples to the server 11, which trains the question-answering model of this application based on the training samples. The trained question-answering model is typically deployed on the server 11 for access and use by the client 12. Alternatively, the data collection tasks performed by the data collection device 10 can be directly completed by the server 11, or the data collection device 10 and the server 11 can be integrated together.

[0050] In one example, as shown in Figure 2, the question-answering model includes a visual encoder, a video frame selector, a Q-former, and a large language model. The process by which client 12 uses the question-answering model to obtain an answer includes: client 12 sends the target question text and the target video to server 11; server 11 encodes each frame in the target video using the visual encoder to obtain the first encoding result for each frame; server 11 uses the video frame selector to obtain the matching score, similarity score, and discriminative score for each frame based on the first encoding result and the target question text; server 11 determines the overall score for each frame based on the matching score, similarity score, and discriminative score; server 11 uses the sampler in the video frame selector to select K first encoding results from all the first encoding results in the target video based on the overall score of each frame; server 11 uses the Q-former to extract features from the K first encoding results to obtain K feature vectors corresponding to the K first encoding results; server 11 uses the large language model to process the K feature vectors and the target question text to obtain the answer to the target question and returns the answer to client 12.

[0051] Referring to Figure 3, Figure 3 is a flowchart illustrating a multimodal question-answering method provided in an embodiment of this application. As shown in Figure 3, the method includes:

[0052] S301, The question-and-answer device acquires the target video and the target question text.

[0053] It should be noted that the question-and-answer device can be server 11 in Figure 1 or a module in server 11.

[0054] The target problem is the problem concerning the target video.

[0055] In one example, the question-answering device receives a question request from a client, which includes a target video and initial information. The initial information can be the target question text, or other information such as video, image, or audio data. If the initial information is other types of information, the question-answering device converts it into the target question text.

[0056] In another example, the target video is stored locally. The question-answering device receives a question request from a client. This request includes first information, which may be the target question text or other information, such as video, image, or audio information. If the first information is other information, the question-answering device converts it into the target question text. The target question text corresponds to the target video, and the question-answering device determines the target video from local storage based on the target question text.

[0057] In another example, the target video is stored locally. The question-answering device receives a question request from a client, which includes initial information and a target video identifier. The initial information can be the target question text or other information, such as video, image, or audio information. If the initial information is other than the target question text, the question-answering device converts it into the target question text. The question-answering device then determines the target video from local storage based on the video identifier.

[0058] S302. The question-and-answer device encodes each frame in the target video to obtain the first encoding result of each frame.

[0059] In one example, the question-answering device uses a visual encoder to encode each frame of the target video to obtain a first encoded result for each frame. The visual encoder can be implemented using one or more transformer layers. In one example, the visual encoder is based on the ViT (vision transformer) model.

[0060] S303, the question-answering device determines at least two of the following scores for each frame in the target video: matching score, similarity score, and discriminative score, based on the first encoding result of each frame and the target question text.

[0061] The matching score for each frame represents the degree of matching between that frame and the target question text; a higher matching score indicates a better match. The similarity score for each frame represents the similarity between that frame and the target question; a higher matching score indicates a better similarity. The distinguishability score for each frame represents the degree of distinguishability between that frame and all other frames in the target video; a higher distinguishability score indicates greater distinguishability, meaning a higher distinguishability score indicates greater importance. In other words, the distinguishability score of each frame can be used to represent its importance relative to all other frames in the target video.

[0062] In one possible implementation, the question-answering device extracts features from the first encoding result of each frame in the target video to obtain a first feature vector for each frame, and encodes the target question text to obtain a second encoding result; the question-answering device calculates a similarity score for each frame based on the first feature vector and the second encoding result of each frame.

[0063] Optionally, before calculating the similarity score for each frame based on the first feature vector and the second encoding result, the question-answering device transforms the first feature vector and the second encoding result of each frame using a linear matrix to obtain the transformed first feature vector and the transformed second encoding result. Subsequently, the question-answering device calculates the similarity score for each frame based on the transformed first feature vector and the transformed second encoding result.

[0064] Optionally, the i-th frame may be preprocessed before encoding. Preprocessing includes, but is not limited to, noise reduction, jitter reduction, and / or deblurring, etc., which are not limited here.

[0065] In one example, the similarity score of the i-th frame in the target video can be characterized as:

[0066] Among them, v i Let be the i-th frame of the target video, q represent the target question text, and E... e and E q These are a feature extractor and a text encoder, used to extract features from the first encoding result of the i-th frame and encode the target question text, respectively. This is the first encoding result for the i-th frame. and h q W represents the first feature vector of the i-th frame and the second encoding result of the target question text, respectively. e and W q It is a linear matrix used to transform the first feature vector and the second encoding result.

[0067] It should be noted that the second encoding result is essentially a vector. The cosine value of the angle between the first feature vector and the second encoding result is calculated by the cosine function, and this cosine value is used as the similarity score of the i-th frame.

[0068] In one possible implementation, the question-answering device performs multimodal fusion encoding on the first encoding result and the target question text of each frame to obtain the third encoding result of each frame; the question-answering device calculates the matching score of each frame based on the third encoding result of each frame.

[0069] Optionally, before calculating the matching score for each frame based on the third encoding result of each frame, the question-answering device transforms the third encoding result using a linear matrix to obtain the transformed third encoding result, and then calculates the matching score for each frame based on the transformed third encoding result.

[0070] In a specific example, the matching score of the i-th frame in the target video can be represented as:

[0071] Among them, W f E is a linear matrix. f This is a multimodal fusion encoder used to perform multimodal fusion encoding on the first encoding result of the i-th frame and the target question text. express The vector corresponding to token[CLS]. The σ function is used to... The calculation is performed, and the result is the matching score of the i-th frame. This is the first encoding result of the i-th frame.

[0072] In one feasible implementation, the question-answering device normalizes each frame in the target video to obtain a normalized result for each frame; it calculates the similarity between the i-th frame and the other video frames in the target video based on the normalized result of the i-th frame and the other video frames in the target video; the question-answering device calculates an average similarity based on the similarity between the i-th frame and the other video frames in the target video; and the question-answering device determines the discrimination score of the i-th frame based on the average similarity.

[0073] In one example, the similarity between frame i and frame j can be represented as:

[0074] Among them, v j For the j-th frame of the target video, and These are the first encoded results of frame i and frame j, respectively. `norm()` is the normalization function used to normalize the encoded data. and Normalization is performed. The cos() function is used to calculate the normalization of the expression. and The cosine value of the angle between the normalized results obtained after normalization is used as the similarity between the i-th frame and the j-th frame.

[0075] The average similarity between frame i and all other frames in the target video except frame i can be expressed as:

[0076] Where M is the number of frames in the target video.

[0077] The discrimination score of the i-th frame can be expressed as:

[0078] In another example, the discrimination score of the i-th frame can be expressed as:

[0079] S304. The question-answering device selects K first coding results from the first coding results corresponding to all frames in the target video based on at least two of the matching score, similarity score and discrimination score of each frame in the target video.

[0080] In one feasible implementation, the question-answering device calculates an overall score for each frame based on the matching score, similarity score, and discrimination score of each frame. The overall score for each frame is the sum or weighted sum of at least two of the matching score, similarity score, and discrimination score. In one example, the overall score of the i-th frame can be expressed as:

[0081] In another example, the overall score for the i-th frame can also be expressed as:

[0082] Where α, β, and γ are weights.

[0083] The question-answering device selects K first-coded results from the first-coded results corresponding to all frames in the target video based on the overall score of each frame. In one example, the question-answering device sorts the first-coded results corresponding to all frames in the target video based on the overall score of each frame, where the higher the overall score of a video frame, the higher its first-coded result ranks in the sorted list; the question-answering device selects the top K first-coded results from the sorted list. In another example, the question-answering device determines the probability of selecting a first-coded result for each frame based on the overall score of each frame, and selects K first-coded results from the first-coded results corresponding to all frames in the target video based on the probability of selecting a first-coded result for each frame in the target video.

[0084] The probability of selecting the first encoding result of the i-th frame can be expressed as:

[0085] Where τ is the temperature coefficient. Give the overall score for frame j.

[0086] S305, The question-answering device obtains the answer to the target question based on K first encoding results and the target question text.

[0087] In one example, the question-answering device extracts features from K first-encoded results to obtain K second-feature vectors; the K second-feature vectors and the target question text are then input into a trained large language model for processing to obtain the answer to the target question.

[0088] It should be noted that the process by which the server processes the target video and the target question text to obtain the answer to the target question is implemented through a question-answering model. As shown in Figure 4, the question-answering model includes a large language model, a visual encoder, a video frame selector, and a Q-former.

[0089] The target video shows a person opening a laptop on a table after drinking water. The target question is: What did the person do while drinking from the cup and touching the table?

[0090] The question-answering device processes each frame of the target video using a visual encoder to obtain a first encoding result for each frame. It then uses a feature extractor to extract features from the first encoding result of each frame to obtain a first feature vector for each frame. Next, it uses a text encoder to encode the target question text to obtain a second encoding result. Based on the first feature vector and the second encoding result of each frame, the question-answering device calculates a similarity score between each frame and the target question text. Finally, it uses a fusion encoder to perform multimodal fusion encoding on the first encoding result and the target question text of each frame to obtain a third encoding result. Based on the vector corresponding to token[CLS] in the third encoding result, it calculates a matching score between each frame and the target question text. Based on the first encoding results corresponding to all frames in the target video, the question-answering device calculates a discrimination score between each frame and all other frames in the target video. Finally, the question-answering device sums or performs a weighted summation of at least two of the similarity score, matching score, and discrimination score for each frame to obtain the overall score for that frame.

[0091] In one example, the question-answering device uses a sampler to sort the first encoded results of all video frames in the target video based on the overall score of all video frames in the target video from high to low, and then selects the top K first encoded results. This method can be called the TopK method.

[0092] In another example, the question-answering device uses sampling to calculate the probability of the first encoded result of each frame being selected based on the overall score of each frame, and selects K first encoded results from the first encoded results of all video frames in the target video based on the probability of the first encoded results being selected in all video frames in the target video.

[0093] It should be noted that the first encoding result selected by TopK may not be the most suitable, or it may be unsuitable. In other words, the suitable first encoding result may not be in TopK, which may lead to the question and answer result obtained based on the selected first encoding result being incorrect. The accuracy of the question and answer result obtained by this method is unstable. However, by using a probabilistic approach, the first encoding result corresponding to all frames in the target video may be selected during frame selection, which helps to improve the stability of the accuracy of the response result obtained by using the solution of this application.

[0094] The question-answering device uses a Q-former to extract features from K first-order encoding results to obtain K second-order feature vectors. Optionally, the question-answering device also uses a Q-former to compress the K second-order feature vectors. The question-answering device then uses a large language model to process the K second-order feature vectors and the target question text to obtain the answer to the target question, such as "opened the laptop".

[0095] Among them, the visual encoder can be implemented by one or more transformers, the feature extractor can be implemented by one or more transformers, the text encoder can be implemented by one or more transformers, and the multimodal fusion encoder can be implemented by one or more transformers.

[0096] The large language model is used to process the K first encoding results and the target question text to obtain the answer to the target question.

[0097] It should be noted that, in addition to the text encoder, fusion encoder and sampler mentioned above, the video frame selector may also include other modules, which are not limited here.

[0098] In another example, the question-answering device uses a feature extractor to extract features from the first encoded result of each frame to obtain the first feature vector of each frame. The device then records the correspondence between the first encoded results and the first feature vectors of all frames in the target video. After determining K first encoded results, the device determines K first feature vectors corresponding to these K first encoded results based on this correspondence. The device then uses a large language model to process the K first feature vectors and the target question text to obtain the answer to the target question. This approach requires feature extraction from the first encoded results again, thus reducing the computational overhead of the question-answering model. Therefore, the Q-former in the question-answering model of this application is optional.

[0099] It can be seen that when selecting video frames for video question answering, selecting the first encoding result strongly related to the target question from the first encoding results corresponding to all video frames in the target video based on at least one of the matching score and similarity score between each frame and the target question text, and then using this first encoding result for subsequent processing, helps improve the accuracy of the answer result. Furthermore, considering the discriminative information of video frames during video question answering ensures that the selected first encoding result contains more information, and subsequent processing based on this first encoding result further improves the accuracy of the answer result. Simultaneously, selecting K first encoding results from the first encoding results corresponding to all frames in the target video based on at least two of the similarity score, matching score, and discriminative score for each frame improves frame selection efficiency, thereby improving question answering efficiency. Moreover, using a probabilistic frame selection method ensures that the first encoding result corresponding to all frames in the target video is likely to be selected, which helps improve the stability of the answer result obtained using the proposed solution. In other words, it helps improve the robustness of the question answering model of this application and avoids overfitting.

[0100] The training process of the question-answering model is described below.

[0101] The training equipment trains the visual encoder and the large language model separately. Then, the parameters of the visual encoder and the large language model are fixed, and the parameters of the video frame selector and the Q-former are trained. The training process is as follows:

[0102] Obtain training samples, which include question text samples, video samples, and answers corresponding to the question text samples. As shown in Figure 4, the training device inputs video samples into a visual encoder for encoding processing to obtain the first encoding result for each frame in the video sample. The training device inputs the first encoding result of each frame into a feature extractor in a video frame selector for processing to obtain the first feature vector of each frame. The question text sample is input into a text encoder in the video frame selector for processing to obtain the second encoding result. Based on the first feature vector and the second encoding result of each frame, the similarity score between each frame in the video sample and the question text sample is calculated. The training device inputs the first encoding result of each frame and the question text sample into a fusion encoder in the video frame selector for multimodal fusion encoding to obtain the third encoding result of each frame. The video frame selector calculates the matching score between each frame and the question text sample based on the third encoding result of each frame. The training device uses the video frame selector to calculate the discrimination score between each frame and all other frames in the video sample, based on the first encoding results corresponding to all frames in the video sample. The server sums or weights at least two of the similarity score, matching score, and discrimination score of each frame to obtain the overall score of each frame.

[0103] The training device uses a sampler to select K first encoding results from the first encoding results corresponding to all frames in the video samples based on the RelaxedTOPK algorithm and the overall score of each frame.

[0104] In one example, based on the RelaxedTOPK algorithm and the overall score of each frame, K first codes are selected from the first code results corresponding to all frames in the video sample. The set of video frames corresponding to these K first codes is as follows: It can be represented as:

[0105] Where S represents the overall score of the video frames in the video sample, V represents the video sample, K represents the number of selected video frames, and τ represents the temperature coefficient.

[0106] During training, the RelaxedTOPK algorithm is used to select frames, so that the gradient can be transmitted to the video frame selector during subsequent backpropagation, thereby achieving the purpose of training the video frame selector.

[0107] The training device uses a Q-former to extract features from K first-order encoding results to obtain K second-order feature vectors. Optionally, the training device also uses a Q-former to compress these K second-order feature vectors. The training device then uses a large language model to process the K second-order feature vectors and the target question text to obtain the predicted answer to the question sample. Based on the predicted answer to the question sample and the corresponding answers to the question samples in the training samples, the training device calculates a loss value. Based on this loss value, it adjusts the parameters in the Q-former and the video frame selector through backpropagation to achieve the purpose of training the Q-former and the video frame selector.

[0108] It should be noted that the above training process can be repeated iteratively multiple times until the loss value calculated in the above manner converges or the number of training iterations reaches the preset number.

[0109] The performance of the question-answering model in this application is described below.

[0110] Table 1 illustrates the performance comparison of the question answering model of this application with the LLaMA-VQA model and the SeViLA model based on three datasets (NExT-QA, STAR, and TVQA).

[0111] Table 1

[0112] As shown in Table 1, the question answering model of this application uses fewer video frames than the LLaMA-VQA model. Furthermore, when verifying the performance of the question answering model, LLaMA-VQA model, and SeViLA model using the three datasets mentioned above, the accuracy of the question answering results obtained by the question answering model of this application is higher than that of the other two models.

[0113] Table 2 shows the ablation ratio results for each score.

[0114] Table 2

[0115] As shown in Table 2, the accuracy of the question-answering results obtained by using frame selection based on three scores (similarity score, matching score, and discrimination score) is higher than that obtained by using two of the above three scores.

[0116] Table 3 illustrates the computational overhead of each module in the two models (unit: 1e-12 times).

[0117] Table 3

[0118] As shown in Table 3, when performing question-and-answer operations, the computational cost of the question-and-answer model proposed in this application is less than that of the SeViLA model. In particular, the video frame selector saves 94% of the floating-point operations. In other words, when performing question-and-answer operations, the efficiency of the question-and-answer model proposed in this application is higher than that of the SeViLA model.

[0119] Table 4 illustrates the performance of the proposed solution before and after combining it with other large language models.

[0120] Table 4

[0121] As shown in Table 4, on the STAR dataset, the Video-LLaMA model incorporating the proposed solution achieves higher accuracy across all four dimensions (interaction, sequence, prediction, and probability) and overall accuracy than the Video-LLaMA model without the proposed solution. Similarly, the BLiP2-Flan-T5-xxL model incorporating the proposed solution also achieves higher accuracy across all four dimensions (interaction, sequence, prediction, and probability) and overall accuracy than the BLiP2-Flan-T5-xxL model without the proposed solution. This demonstrates that the proposed solution can be quickly integrated with various large language models and improve model performance.

[0122] Referring to Figure 5, a structural schematic diagram of a question-and-answer device provided in an embodiment of this application is shown. As shown in Figure 5, the question-and-answer device 500 includes:

[0123] Acquisition unit 501 is used to acquire the target video and the target question text;

[0124] The encoding unit 502 is used to encode each frame in the target video to obtain the first encoding result of each frame;

[0125] The determining unit 503 is used to determine at least two of the following for each frame: similarity score, matching score, and discrimination score, based on the first encoding result of each frame in the target video and the target question text: the similarity score is used to characterize the degree of similarity between each frame and the target question, the matching score is used to characterize the degree of matching between each frame and the target question, and the discrimination score is used to characterize the degree of discrimination between each frame and other video frames in the target video other than each frame.

[0126] Selection unit 504 is used to select K first encoding results from the first encoding results corresponding to all frames in the target video based on at least two of the similarity score, matching score and discrimination score of each frame; where K is an integer greater than 1;

[0127] The determining unit 503 is also used to obtain the answer to the target question based on the K first encoding results and the target question text.

[0128] In one feasible implementation, unit 504 is specifically used for:

[0129] The overall score of each frame is calculated based on at least two of the similarity score, matching score, and discrimination score. The first encoding results of the video frames in the target video are sorted from high to low based on the overall score of each frame, and the top K first encoding results are selected from the sorted results.

[0130] In one feasible implementation, unit 504 is specifically used for:

[0131] The overall score of each frame is calculated based on at least two of the similarity score, matching score, and discriminative score. The selection probability of each frame is determined based on the overall score of each frame. The selection probability of each frame is the probability that the first encoded result of each frame is selected from the first encoded results corresponding to all frames in the target video. K first encoded results are selected from the first encoded results corresponding to all frames in the target video based on the selection probability of each frame.

[0132] In one feasible implementation, the determining unit 503 is specifically used for:

[0133] Feature extraction is performed on the first encoding result of each frame to obtain the first feature vector of each frame; the target question text is encoded to obtain the second encoding result; the similarity score of each frame is calculated based on the first feature vector and the second encoding result of each frame; multimodal fusion encoding is performed on the first encoding result and the target question text of each frame to obtain the third encoding result corresponding to each frame; the matching score of each frame is calculated based on the third encoding result corresponding to each frame; the first encoding result of each frame is normalized to obtain the normalized result of each frame; the discriminative score of each frame in the target video is calculated based on the normalized result of all video frames in the target video.

[0134] In one feasible implementation, in calculating the discriminative score of each frame in the target video based on the normalized processing results of all video frames in the target video, the determining unit 503 is specifically used for:

[0135] The similarity between frame i and each of the target video frames other than frame i is calculated based on the normalized result of frame i and the normalized result of each of the target video frames other than frame i. The average similarity between frame i and each of the target video frames other than frame i is calculated. The discrimination score of frame i is determined based on the average value. The discrimination score of each frame in the target video includes the discrimination score of frame i.

[0136] It is worth noting that the specific functional implementation of the question-answering device 500 is described in the relevant description of the multimodal question-answering method above. For example, the acquisition unit 501 is used to execute the relevant content of S301, the encoding unit 502 is used to execute the relevant content of S302, the determination unit 503 is used to execute the relevant content of S303 and S305, and the selection unit 504 is used to execute the relevant content of S304. Each unit or module in the question-answering device 500 can be individually or entirely merged into one or more other units or modules, or some of the units or modules can be further divided into multiple functionally smaller units or modules. This achieves the same operation without affecting the technical effect of the embodiments of this application. The above-mentioned units or modules are based on logical function division. In practical applications, the function of one unit (or module) is implemented by multiple units (or modules), or the function of multiple units (or modules) is implemented by one unit (or module).

[0137] Based on the description of the above method embodiments and related device embodiments, please refer to Figure 6, which is a schematic diagram of the structure of another question-answering device 600 provided in this application embodiment. The question-answering device 600 shown in Figure 6 includes a memory 601, a processor 602, a communication interface 603, and a bus 604. The memory 601, processor 602, and communication interface 603 are interconnected via the bus 604.

[0138] Optionally, the memory 601 can be ROM, static storage device, dynamic storage device, or RAM.

[0139] The memory 601 is capable of storing programs. When the programs stored in the memory 601 are executed by the processor 602, the processor 602 and the communication interface 603 are used to execute the various steps of the multimodal question-answering method of the embodiment shown in FIG3.

[0140] The processor 602 employs a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), GPU, or one or more integrated circuits to execute relevant programs to implement the multimodal question-answering method of the embodiment shown in Figure 3.

[0141] The processor 602 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the multimodal question-answering method of this application can be completed through integrated logic circuits in the hardware of the processor 602 or instructions in software form. Optionally, the processor 602 is a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor is a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. Optional software modules are located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory 601. The processor 602 reads the information in the memory 601 and, in conjunction with its hardware, performs the functions required by the units included in the question-answering device 500 of this application embodiment, or executes the multimodal question-answering method of the method embodiment of this application.

[0142] The communication interface 603 uses transceiver-related devices, such as, but not limited to, transceivers, to enable communication between the question-and-answer device 600 and other devices or communication networks.

[0143] Bus 604 may include a pathway for transmitting information between various components of question-and-answer device 600 (e.g., memory 601, processor 602, communication interface 603).

[0144] It should be noted that although the question-answering device 600 shown in Figure 6 only illustrates the memory, processor, and communication interface, those skilled in the art should understand that in specific implementations, the question-answering device 600 also includes other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the question-answering device 600 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the question-answering device 600 may only include the devices necessary for implementing the embodiments of this application, and not necessarily all the devices shown in Figure 6.

[0145] This application also provides a chip, which includes a processor and a data interface. The processor reads instructions stored in a memory through the data interface to implement the multimodal question answering method and related methods of this application.

[0146] Optionally, as one implementation, the chip further includes a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the multimodal question answering method and related methods.

[0147] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform one or more steps of any of the above methods.

[0148] This application also provides a computer program product containing instructions. When the computer program product is run on a computer or processor, it causes the computer or processor to perform one or more steps of any of the methods described above.

[0149] Those skilled in the art will appreciate that the functionality described in conjunction with the various illustrative logic blocks, modules, and algorithmic steps disclosed herein can be implemented by hardware, software, firmware, or any combination thereof. If implemented in software, the functionality described by the various illustrative logic blocks, modules, and steps can be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may comprise a computer-readable storage medium, which corresponds to a tangible medium, such as a data storage medium, or a communication medium that includes any medium facilitating the transfer of a computer program from one place to another (e.g., based on a communication protocol). In this way, the computer-readable medium may substantially correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium, such as a signal or carrier wave. The data storage medium may be any available medium accessible by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this application. A computer program product may comprise a computer-readable medium.

[0150] By way of example and not limitation, such computer-readable storage media includes RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other media that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer. Furthermore, any connection is properly referred to as computer-readable media. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. However, it should be understood that the computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other temporary media, but are specifically referring to non-temporary tangible storage media. As used herein, disks and optical discs include Compact Discs (CDs), Laser Discs, Optical Discs, Digital Versatile Discs (DVDs), and Blu-ray Discs, where disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of these should also be included within the scope of computer-readable media.

[0151] Instructions can be executed by one or more processors, such as one or more DSPs, general-purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other structures suitable for implementing the techniques described herein. Furthermore, in some aspects, the functions described in the various illustrative logic blocks, modules, and steps described herein are provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Moreover, the techniques can be fully implemented within one or more circuit or logic elements.

[0152] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the division of units is merely a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Optionally, the coupling, direct coupling, or communication connection shown or discussed between them may be through some interfaces, indirect coupling or communication connection of devices or units, such as electrical, mechanical, or other forms.

[0153] Optionally, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0154] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of this application are generated.

[0155] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.

Claims

1. A multimodal question-answering method, characterized in that, include: Obtain the target video and the target question text; Each frame in the target video is encoded to obtain a first encoding result for each frame; Based on the first encoding result of each frame in the target video and the target question text, at least two of the following are determined for each frame: similarity score, matching score, and discrimination score. The similarity score is used to characterize the degree of similarity between each frame and the target question, the matching score is used to characterize the degree of matching between each frame and the target question, and the discrimination score is used to characterize the degree of discrimination between each frame and other video frames in the target video. Based on at least two of the similarity score, matching score, and discriminative score of each frame, K first encoding results are selected from the first encoding results corresponding to all frames in the target video; wherein, K is an integer greater than 1; The answer to the target question is obtained based on the K first encoding results and the target question text.

2. The method according to claim 1, characterized in that, The selection of K first encoding results from the first encoding results corresponding to all frames in the target video based on at least two of the similarity score, matching score, and discriminative score of each frame includes: The overall score of each frame is calculated based on at least two of the similarity score, matching score, and discrimination score of each frame. Based on the overall score of each frame, the first encoding results of all frames in the target video are sorted from high to low, and the top K first encoding results are selected from the sorted results.

3. The method according to claim 1, characterized in that, The selection of K first encoding results from the first encoding results corresponding to all frames in the target video based on at least two of the similarity score, matching score, and discriminative score of each frame includes: The overall score for each frame is calculated based on at least two of the similarity score, matching score, and discrimination score of each frame. The selection probability of each frame is determined based on the overall score of each frame, and the selection probability of each frame is the probability that the first encoding result of each frame is selected from the first encoding results corresponding to all frames in the target video. Based on the selection probability of each frame, K first encoding results are selected from the first encoding results corresponding to all frames in the target video.

4. The method according to any one of claims 1-3, characterized in that, The determination of the similarity score, matching score, and discriminative score of each frame in the target video based on the first encoding result of each frame in the target video and the target question text includes: Feature extraction is performed on the first encoding result of each frame to obtain the first feature vector of each frame; the target question text is encoded to obtain the second encoding result; The similarity score of each frame is calculated based on the first feature vector and the second encoding result of each frame; Multimodal fusion encoding is performed on the first encoding result of each frame and the target question text to obtain the third encoding result corresponding to each frame; the matching score of each frame is calculated based on the third encoding result corresponding to each frame. The first encoding result of each frame is normalized to obtain the normalized result of each frame; The discrimination score of each frame in the target video is calculated based on the normalization results of all video frames in the target video.

5. The method according to claim 4, characterized in that, The discrimination score for each frame in the target video is calculated based on the normalization results of all video frames in the target video, including: The similarity between the i-th frame and the other video frames in the target video (excluding the i-th frame) is calculated based on the normalized result of the i-th frame and the normalized result of the other video frames in the target video (excluding the i-th frame). The average similarity between the i-th frame and each of the other video frames in the target video, excluding the i-th frame, is calculated. The discrimination score of the i-th frame is determined based on the average value, and the discrimination score of each frame in the target video includes the discrimination score of the i-th frame.

6. A question-and-answer device, characterized in that, include: The acquisition unit is used to acquire the target video and the target question text. An encoding unit is used to encode each frame in the target video to obtain a first encoding result for each frame; A determining unit is configured to determine at least two of the following for each frame based on the first encoding result of each frame and the target question text: a similarity score, a matching score, and a distinguishing score. The similarity score is used to characterize the degree of similarity between each frame and the target question, the matching score is used to characterize the degree of matching between each frame and the target question, and the distinguishing score is used to characterize the degree of distinguishing between each frame and other video frames in the target video. A selection unit is configured to select K first encoding results from the first encoding results corresponding to all frames of the target video based on at least two of the similarity score, matching score, and distinguishability score of each frame; wherein K is an integer greater than 1; The determining unit is further configured to obtain the answer to the target question based on the K first encoding results and the target question text.

7. The question-and-answer device according to claim 6, characterized in that, The selection unit is specifically used for: The overall score of each frame is calculated based on at least two of the similarity score, matching score, and discrimination score of each frame. Based on the overall score of each frame, the first encoding results of all frames in the target video are sorted from high to low, and the top K first encoding results are selected from the sorted results.

8. The question-and-answer device according to claim 6, characterized in that, The selection unit is specifically used for: The overall score for each frame is calculated based on at least two of the similarity score, matching score, and discrimination score. The selection probability of each frame is determined based on the overall score of each frame, and the selection probability of each frame is the probability that the first encoding result of each frame is selected from the first encoding results corresponding to all frames in the target video. Based on the selection probability of each frame, K first encoding results are selected from the first encoding results corresponding to all frames in the target video.

9. The question-and-answer device according to any one of claims 6-8, characterized in that, The determination of the similarity score, matching score, and discriminative score for each frame based on the first encoding result of each frame and the target question text includes: Feature extraction is performed on the first encoding result of each frame to obtain the first feature vector of each frame; the target question text is encoded to obtain the second encoding result; The similarity score of each frame is calculated based on the first feature vector and the second encoding result of each frame; Multimodal fusion encoding is performed on the first encoding result of each frame and the target question text to obtain the third encoding result corresponding to each frame; the matching degree score of each frame is calculated based on the third encoding result corresponding to each frame. The first encoding result of each frame is normalized to obtain the normalized result of each frame; The discrimination score of each frame in the target video is calculated based on the normalization results of all video frames in the target video.

10. The question-and-answer device according to claim 6, characterized in that, In the aspect of calculating the discriminative score of each frame in the target video based on the normalized processing results of all video frames in the target video, the determining unit is specifically used for: The similarity between the i-th frame and the videos in the target video other than the i-th frame is calculated based on the normalized processing result of the i-th frame and the normalized processing results of the other videos in the target video. The average similarity between the i-th frame and each of the other video frames in the target video, excluding the i-th frame, is calculated. The discrimination score of the i-th frame is determined based on the average value, and the discrimination score of each frame in the target video includes the discrimination score of the i-th frame.

11. A question-and-answer device, characterized in that, The method includes a processor and a memory, wherein the memory is used to store program code, and the processor is used to execute the program code to implement the method according to any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-5.

13. A computer program product that, when run on a computer, causes the computer to perform the method as described in any one of claims 1-5.

Images