Multi-modal retrieval prompt learning

Abstract

Described is a method for multi-modal retrieval prompt learning, the method comprising: receiving a dataset relating to a single query, the dataset comprising one or more positive sample and one or more negative sample; generating one or more pair of samples using the received dataset, each pair of samples comprised of one positive sample and one negative sample; encoding each of the one or more pair of samples; calculating a loss function for each encoded pair of samples based on a comparison of the encoded positive sample and the encoded negative sample to a learnt encoded query prompt; generating a learnt prompt for retrieval by optimising the calculated loss function; and outputting the learnt prompt.

Description

[0001] MULTI-MODAL RETRIEVAL PROMPT LEARNING

[0002] TECHNICAL FIELD

[0003] The present disclosure relates to a method and apparatus for performing prompt optimisation in a multi-modal neural network.

[0004] BACKGROUND

[0005] The natural world consists of diverse matter properties, including structure, size, spectral information (e.g., colour), and inertial data (direction, speed, acceleration), and other distinct categories of information. Both humans and machines perceive and interpret this through various sensors — like eyes, ears, cameras, and microphones - each representing a unique modality. These modalities vary widely in their raw data formats and the types of information they convey, such as visual items and properties, depth, thermal, inertial, auditory, tempo and even emotion. The human brain integrates inputs from these modalities into a unified format, aiding comprehension of complex data.

[0006] Advances in Deep Neural Networks (DNNs) and Artificial Intelligence (Al) have revolutionized information processing across modalities. Early DNNs focused on single-modal tasks like vision, enabling classification and detection using pre-trained networks and large datasets to learn similarities among predefined labels. The evolution to the second generation introduced semi-supervised representation learning techniques, compressing inputs into latent representations that support similarity searches, allowing for efficient massive storage in database systems and retrieval of relevant items (also known as query or search).

[0007] More recently, Contrastive Multi-Modal Neural Networks (CMMNN), exemplified by works like "CLIP: Connecting Text and Images" (2021, Radford et al.), integrate multiple modalities into a unified multi-dimensional space, thus imbue a semantic structure into the latent embedding space. This enables zero-shot similarity inference across modalities, allowing inputs from one modality (e.g., text) to compare, classify and even retrieval of relevant items of another modality (e.g., vision) stored in a large-scale database.

[0008] Despite these advancements, the diverse nature of the natural world and the specific contextual needs of different industries and applications pose ongoing challenges in improving the accuracy of multi-modal retrieval.

[0009] Practical Challenges: Achieving high retrieval accuracy presents practical challenges. Variability in input types, including implicit or semantic search queries and "long-tail" scenarios not covered during CMMNN training (e.g., concepts like "broken" or complex situations like "illegal parking"), can limit performance.

[0010] Existing solutions: Current approaches propose two main strategies: (A) Model Fine-tuning, which requires extensive data collection tailored to specific cases of interest, and (B) Few-shot Learning (FSL), via prompt-tuning (or adapters on VLMs). FSL has the advantages of requiring relatively little data (and correspondingly, computation), being intuitively suitable for learning new concepts thus addressing the long-tail problem, and unlike model fine-tuning it is inherently “local”, i.e., it can optimize few weights per-query instead of many weights on many queries. However, existing FSL literature in the context of VLMs always addresses the “k-shot n-way” classification problem. It is not immediately clear how to adapt these solutions to the retrieval problem.

[0011] The present disclosure is therefore aimed towards providing methods and systems for solving at least the afore-mentioned problems. SUMMARY

[0012] According to a first aspect of this disclosure there is provided a method for multi-modal retrieval prompt learning, the method comprising: receiving a dataset relating to a single query, the dataset comprising one or more positive sample and one or more negative sample; generating one or more pair of samples using the received dataset, each pair of samples comprised of one positive sample and one negative sample; encoding each of the one or more pair of samples; calculating a loss function for each encoded pair of samples based on a comparison of the encoded positive sample and the encoded negative sample to a learnt encoded query prompt; generating a learnt prompt for retrieval by optimising the calculated loss function; and outputting the learnt prompt. This allows for increased retrieval accuracy and low computation cost as each query prompt is learnt independently.

[0013] The method as described above, wherein the learnt query prompt represents the current learnt prompt state. This allows for an initial learnt prompt to be used as the starting point for optimisation.

[0014] The method as described above, wherein the comparison of the encoded positive sample and the encoded negative sample to the learnt encoded query prompt comprises calculating cosine similarity between each of the encoded positive sample and the encoded negative sample with the learnt encoded query prompt. This allows for the penalisation to be determined based on cosine similarity.

[0015] The method as described above, wherein the method further comprises calculating the loss function using a model; and wherein the method further comprises updating the learnt encoded query prompt when the cosine similarity between the encoded negative sample and the learnt encoded query prompt is greater than the cosine similarity between the encoded positive sample and the learnt encoded query prompt. This allows the learnt encoded query prompt to be further refined based on proximity to received encoded samples.

[0016] The method as described above, wherein the method further comprises the step of storing the output learnt prompt; and retrieving data items from an embeddings database using the learnt prompt. This allows for the most relevant stored data items to be returned based on the learnt prompt.

[0017] The method as described above, wherein the calculated loss function comprises calculating a marginized binary cross entropy loss function. This provides a first modified method of generating a loss function.

[0018] The method as described above, wherein the marginized binary cross entropy loss function is calculated based on the expression:

[0019]

[0020] where y is the number between 1 and 0 representing the specific sample relevance, whether it’s value for a positive sample is 1, and its value for a negative sample is 0, and its value for the intermediate sample number between 1 and 0 which is controllable hyperparameter. Additionally, the p is an output number between 0 and 1. This provides the specific implementation of the modified first loss function.

[0021] The method as described above, wherein the calculated loss function comprises calculating a triplet loss function. This provides a second modified method of generating a loss function.

[0022] The method as described above, wherein the triplet loss function is calculated based on the expression: max(Dp — Dn + M, 0)

[0023] where Dp is the Euclidian distance between the (embedding of the) positive example and the anchor, Dn is the distance between the negative example and the anchor, and M>0 is a margin hyperparameter. This provides the specific implementation of the modified second loss function.

[0024] The method as described above, wherein the calculated loss function comprises calculating a pairwise ranking loss function. This provides a third modified method of generating a loss function.

[0025] The method as described above, wherein the pairwise ranking loss function is calculated based on the expression:

[0026] max(Sn — Sp + M , 0)

[0027] where Sn is the Cosine similarity score of the negative example, Sp is the cosine similarity score of the positive example, and M>0 is a margin hyperparameter. This provides the specific implementation of the modified third loss function.

[0028] The method as described above, wherein the calculated loss function is supplemented by applying a Kullback-Leibler loss function. This provides an additional modification to the employed loss function to provide further accuracy.

[0029] The method as described above, wherein the Kullback-Leibler loss function is calculated based on the expression:

[0030]

[0031] A method of determining query relevance in a multimodal system, the method comprising: receiving a dataset relating to a query prompt, the dataset comprising one or more positive sample, one or more intermediate sample and one or more negative sample; generating two or more pair of samples using the received dataset, each pair of samples comprised of either one positive sample and one negative sample, one positive sample and one intermediate sample, or one negative sample and one intermediate sample; encoding each of the one or more pair of samples; calculating a loss function for each encoded pair of samples; providing a margin coefficient for representing a weighting of the relevance of each of the one or more pairs; adapting the query prompt based on the provided margin coefficient. This introduces a measure of the relevance of an image into the determination of query prompt relevance and therefore allows for increased prompt optimisation.

[0032] The method as described above, wherein the one or more intermediate sample represents a negative sample that is relevant to the query prompt. This provides increased accuracy of the prompt optimisation.

[0033] The method as described above, wherein the margin coefficient represents a controllable value assigned to each pair of samples; wherein the margin coefficient is controlled via a hyper parameter. This provides user control of the weighting applied to each of the types of samples.

[0034] The method as described above, wherein query prompt is of a modality type that comprises text, images, audio, sensor data, LiDAR data, 3D visual data and / or video, logic data, transaction data, entity profiling data or state data. This allows for a number of different modalities to be utilised as part of the method and system. There is also provided a system configured to perform multi-modal retrieval prompt learning, the system comprising one or more processors configured to perform the method according to any preceding claim. This provides a system capable of performing the method of this disclosure.

[0035] The method as described above, wherein the method is performed by a neural network. This allows for a specific implementation of the method.

[0036] There is also provided a computer program stored in stored in non-transitory form comprising a program code for performing the method as described above when executed on a computer. This provides a computer program capable of implementing the method of this disclosure.

[0037] BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The present disclosure is described by way of example, with reference to the accompanying drawings, in which:

[0039] Fig. 1 illustrates an example of a Multi-Modal storage and retrieval system in which the present disclosure optimises the prompt embedding B:

[0040] Fig. 2 shows a further example of a Multi-Modal storage and retrieval system;

[0041] Fig. 3 shows an N-way-K-shot Classification method;

[0042] Fig. 4 illustrates a method and system for Multi-Modal Retrieval Prompt Learning;

[0043] Fig. 5 illustrates a method and system for Multi-Modal Retrieval Prompt Learning including a retrieval stage;

[0044] Fig. 6 illustrates an example of the implementation of marginized Binary Cross Entropy loss;

[0045] Fig. 7 illustrates an example of the implementation of Triplet Loss;

[0046] Fig. 8 illustrates an example of the implementation of Pairwise Ranking loss;

[0047] Fig. 9 illustrates an example of supplementing the calculated the loss function;

[0048] Fig. 10 illustrates an example of sample pairings and their associated controllable margin coefficients in a tier margin implementation.

[0049] DETAILED DESCRIPTION

[0050] In the field of artificial intelligence (Al), a fundamental technology is the Deep Neural Network (DNN). DNNs are used to create various purpose-built models, including both "general-purpose" and highly specialized ones, collectively referred to as "models." A well-known model type is the “classification” model, which assigns a category or “class” to input data (e.g., classifying an image as a “dog” or “cat”). Another important model type is “generative,” such as a large language model (LLM) that generates text in response to text inputs.

[0051] An additional significant model type is an “encoder,” which encodes a large data item into a shorter representation called an "embedding". This process is analogous to lossy compression, resulting in a condensed vector representation that captures the data's semantic meaning. In such representations, similar meanings appear closer together in vector space. For instance, words with similar meanings are positioned closer in the embedding space, enabling efficient comparisons between embeddings to achieve complex semantic evaluations between large data items.

[0052] Embeddings are a key technology that enables the encoding, indexing, storage, and efficient retrieval (search) of specific items from large databases.

[0053] Multimodal Learning (also referred to as multi-modal learning) is a deep learning approach that integrates and processes multiple data types, known as modalities (e.g., text, audio, images, or video). This integration supports a holistic understanding of complex data, improving model performance in tasks like visual question answering, cross-modal retrieval, text-to-image generation, aesthetic ranking, and image captioning.

[0054] A Visual Language Model (VLM) Encoder is an example of multimodal model, specifically a dual-modality model, that processes either text or image inputs to produce a fixed-size embedding. During training, the model aligns the representations of text and image modalities, enabling comparisons between different data types. For example, vector similarity calculations — such as “dot product” or “cosine similarity” can then be used to measure similarity between embeddings.

[0055] This combination of techniques allows for the storage of vast numbers of images in a database and facilitates efficient ad-hoc searches. During a search, a query (known as a "prompt") from either text or image modality is converted into an embedding and compared with stored embeddings in the database. The database then retrieves the top-K most relevant items. For example, a text prompt like “car” may retrieve images of cars. This process of accurate search and retrieval is referred to as “retrieval.” Due to the ambiguous nature of different modalities, various prompts may refer to the same concept. For instance, “car” could also be referred to as “motor car,” “vehicle,” “machine,” “auto,” or “automobile.” Similarly, an image may contain multiple cars or show cars along with other objects, making it unclear which prompt best represents a specific object or scene.

[0056] In model training, "longtail" (or long-tail) data represents the less common cases or edge scenarios that the model must learn to handle, ensuring better generalization and accuracy across diverse scenarios.

[0057] Prompt-tuning (or prompt-engineering) is the process of crafting a query that an Al model can interpret and respond to, whether the model is generative or contrastive.

[0058] There are a number of known techniques that focus on model specialisation such as Model Fine Tuning and Model Prompt Tuning. Such techniques will now be described.

[0059] The Model Fine-Tuning approach requires extensive sample collection tailored to specific cases of interest, significant computational resources, and time for model training (fine-tuning). Some examples of model fine tuning can be found in “Robust Fine-Tuning of Zero-Shot Models” by Wortsman et al, 2022 and “Don't Stop Learning: Towards Continual Learning for the CLIP Model” by Ding et al, 2022. However, this approach has several disadvantages such as that it requires extensive collection (sometimes massive data) tailored to specific cases of interest; or that the model is susceptible to catastrophic forgetting, where previously learned cases may experience reduced accuracy over time. New cases usually require global retraining.

[0060] An alternate technique is few-shot Learning (FSL ) using Prompt-Tuning (or adapters) for Classification: This approach requires fewer samples, easing its use and cost-effectiveness. Being local, this technique also solves the catastrophic forgetting problem. It is used for N-way-K-shot classification, where all classes (“way”) are learned simultaneously, resulting in a single learned vector. This vector is combined, as a prefix, with class names to create final N prompt- vectors. For image classification, all vectors are used, and cosine similarity determines the class, with the highest score indicating the class and confidence level. Some examples of few-shot learning can be found in “CoOp, Learning to Prompt for Vision-Language Models” by Zhou et al, 2021, “ProGRAD, Prompt-aligned Gradient for Prompt Tuning” by Zhu et al, 2022, “MaPLe: Multi-modal Prompt Learning” by Khattak et al, 2022 and “CLIP-Adapter: Better Vision-Language Models with Feature Adapters” by Gao et al, 2023. However, the disadvantage with this approach is that it is not known how to apply this approach to a query retrieval scenario that utilises a single query / class.

[0061] To explain further, an N-way-K-shot Classification aims to classify an unseen sample into one of N classes after being given N*K examples. Importantly, this requires at least two classes (N>1), or it is not possible to perform the classification. An example, of this can be seen in Fig. 3.

[0062] In retrieval contexts, there is typically a single query, such as "uncovered manhole." One can try to naively reduce the problem into a binary classification one by generating classes according to the event type. For example, if the event is “uncovered manhole”, we may classify “uncovered manhole” vs. “covered manhole”. But this ‘translation’ becomes unwieldly. The real class that is desired to be found, given a query X, is “not X”, however, this is not well-defined for a CMMNN such as a VLM. In addition, these methods lack the option to prioritize “hard-negatives” (HN) over “easy negatives” (EN). This problem is especially relevant for the retrieval scenario where users prefer to see correct items (“hard positives”, HP), then the incorrect but relevant (HN), and at last all the rest (EN). For example, for a search of “covered manhole” retrieve first those (HP), then the “uncovered manhole” (HN), and at last anything else (EN).

[0063] A further technique that could be used is Prompt-Tuning for generative models, such as LLM and VLLM which aim to enhance how models generate complex answers (text or other modalities) by improving their understanding of specific questions or requests. However, a disadvantage of this approach is that this method is not suitable for database retrieval tasks, as not dealing contrastive and embedding generating models.

[0064] The problems therefore to be solved by this disclosure are how to perform a MM Prompt-Learning with K-shots and One-Way (1 query / prompt) fitting the CMMNN models in a retrieval scenario (vs. previous scenarios dealing with classification or generative models and scenarios). A further problem solved by the methods and apparatus of this disclosure is the introduction of Learning Loss methods required for MM Retrieval Prompt Learning, some used as is and others with specific adaptations to support the hereby innovation. This disclosure also introduces a way to control the level of relevance between multimodal items, such as (but not limited to): a) hard positives (HP), b) hard-negatives (HN), and c) unrelated easy negatives (EN) The present disclosure addresses the optimization of prompt-tuning accuracy within a retrieval context, through a promptlearning technique that leverages one sample from modality A and a few samples from modality B.

[0065] Fig. 1 illustrates an example of a Multi-Modal storage and retrieval system in which the present disclosure optimises the prompt embedding B. As can be seen in Fig. 1 the left-hand side of the figure represents the data encoding stage and the right-hand side of the figure represents the retrieval stage. Data of modality A 110 is encoded by a multimodal encoder 111 to form embedding A 112 which is stored in a MM database 113. A user may then generate a query 116 that will be encoded by and encoder 115 to form embedding B 114. The embedding B 114 will then be compared to the stored embeddings in order to determine cosine similarity and return the top-k relevant results to the query. Fig. 2 illustrates a similar example in which retrieval is performed based on multiple modalities.

[0066] Fig. 3 illustrates a data classification method in which data of a first modality may be received and encoded by a MM encoder to form embedding A. Data of a second modality may also be received and encoded using a MM encoder to form an embedding B. Multiple embeddings may be provided for each modality of data. These embeddings may be compared to each other for cosine similarity in order to generate a confidence of match between the embeddings. This confidence factor may then be used to determine whether there is a match.

[0067] Standard N-way-K-shot classification aims to classify an unseen sample into one of N classes after being given N*K examples. Importantly, this requires at least two classes (N>1), or there’s nothing to classify. In retrieval contexts, there is typically a single query, such as "uncovered manhole." One can try to naively reduce the problem into a binary classification one by generating classes according to the event type. For example, if the event is “uncovered manhole”, we may classify “uncovered manhole” vs. “covered manhole”. But this ‘translation’ becomes unwieldly. The real class we’re looking for, given a query X, is “not X”. But this is not well-defined for a VLM.

[0068] Fig. 4 illustrates a method and system for Multi-Modal Retrieval Prompt Learning that addresses this problem. This is achieved generally by, receiving a dataset relating to a single query, the dataset comprising one or more positive sample and one or more negative sample. The method then comprises generating one or more pair of samples using the received dataset, each pair of samples comprised of one positive sample and one negative sample; and encoding each of the one or more pair of samples. The method may then calculate a loss function for each encoded pair of samples based on a comparison of the encoded positive sample and the encoded negative sample to a learnt encoded query prompt. The method may also comprise generating a learnt prompt for retrieval by optimising the calculated loss function; and outputting the learnt prompt.

[0069] In other words, the example of Fig. 4 discloses constructing binary pairs of samples (containing one positive and one negative sample), and designing one or more loss function / s to make the learned query prompt embedding (LQPe) more similar to the positive sample embedding (PSe) and less similar to the negative sample embedding (NSe). The model is penalized if it assigns a higher similarity score to the negative image than to the positive one. This penalization may take place when generating the learnt prompt which may be an updated version of an initial learnt prompt. The comparison may be performed using cosine similarity or other methods as will be apparent from this disclosure.

[0070] Fig. 5 illustrates an example embodiment similar to Fig. 4, with the addition of a retrieval stage. Fig. 5 comprises two stages, the first stage being Multi-Modal (MM) Prompt-Learning stage, for generation of an “Retrieval Learnt Prompt”, which may be an embedding vector aligned to modalities of the specific CMMNN encoding model in use. The second stage may be the retrieval stage, which uses the already learnt prompt (and saved on disk or elsewhere), in order to access the MM Embeddings database and retrieve top-k relevant data items.

[0071] The following describes an example of the process of the MM Prompt-Learning stage, which includes receiving an input, which mat comprise the query in Modality 1 (i.e. text), and one or more samples of modality 2 of n-Tiers, each k-Shots (samples). The tiers denote relevance to the query, hard positives to easy negatives. The method may also include dataset preparation comprising Pairwise Cross-Product on Ranks. This may be achieved by producing sample pairs, where each pair may have a Tier Ni shot and Tier Nj shot, where i<j represents different tiers. In some examples, the Learned Prompt may be composed from the (fixed) embedding of Query of modality 1 (i.e. text), concatenated onto the “soft prompt” of 16x265 learnable weights. The method of Fig. 5 may further include a learning step. In the learning step the method may comprise sampling batches of pairs (such as images) and, per pair of samples (ij), computing embeddings and scores for both images in pair as well as computing triplet loss or ranking loss on the pair. The learning step may further calculate gradients on the soft prompts and step the optimizer as well as output features of the MM-Retrieval Leamt-Prompt. This learnt prompt can then be retrieved as part of the retrieval stage.

[0072] There are a number of different methods that can be sued to generate / calculate the loss in the method of this disclosure, some examples of which will now be described. A first loss function that may be utilised is Binary Cross Entropy Loss as illustrated by Fig. 6. Fig. 6 illustrates how similarity score is adapted by stretching the limits . A single similarity score between query and image is sufficient in order to learn from binary classification data. The method of generating loss may comprise translating the similarity score into a Bernoulli distribution, and then utilising cross entropy loss.

[0073] More specifically, for a similarity score x, it is possible to compute the parameter p given by the expression,

[0074]

[0075] where a and b are learned parameters and S(x) is the Sigmoid function which maps R to (0,1 ). The output p is a number between 0 and 1, representing the Bernoulli distribution (l-p,p), i.e., how likely the image is to be positive with respect to the query. The parameters a and b are learned to bridge between the output contrastive similarity scores aligned to the multi-modal model such as BLIP (0.1-0.5) and the input space of the sigmoid (note, S(0)=l / 2). Therefore, Given p, it is possible to compute the binary cross entropy for label y using the expression:

[0076]

[0077] < >

[0078] Where y is the number between 1 and 0 representing the specific sample relevance, whether it’s value for a positive sample is 1 , and its value for a negative sample is 0, and its value for the intermediate sample number between 1 and 0 which is controllable hyperparameter. Additionally, the p is an output number between 0 and 1. Alternatively, Marginized Binary Cross Entropy may be utilised instead of the standard binary cross entropy discussed above. Binary cross-entropy loss is binary by nature, it is concerned only with distinguishing between “correct” and “incorrect” items, leading to y=l or y=0. The Marginized binary cross-entropy of this disclosure relaxes this constraint by supporting a 0>y<l enabling this technique to learn from the differences between items of different relevance levels, and prioritizing incorrect-but-relevant items (hard negatives) over incorrect (easy negatives).

[0079] An alternate method for calculating the loss as part of the method of this disclosure is the use of Triplet Loss. An example of a method that utilises triplet loss can be seen in Fig. 7. Triplet loss forces the separation between different pairings by a specified margin value, where related data points are projected near each other. In contrast, disparate data points are projected far away. This disclosure adapts the formulation of triplet loss to provide an anchor in the prompt tuned text, a positive that may be image similar to text, and a negative that may be image not-similar to text. The use of Multi-Modal aligned encodings (such as CLIP-like) can be assumed such that embeddings of samples (such as image) and query (such as text) are in the same space. The triplet loss of this disclosure may therefore be represented by the following expression,

[0080] max(Dp — Dn + M, 0)

[0081] where Dp is the Euclidian distance between the (embedding of the) positive example and the anchor, Dn is the distance between the negative example and the anchor, and M>0 is a margin hyperparameter.

[0082] A further example of how the loss function may be calculated in the present disclosure can be seen in Fig. 8, in which cosine similarity may be utilised between the positive sample, the negative sample and the anchor tuned prompt. Fig. 8 illustrates a pairwise ranking loss (PWR). In such an approach, an approach such as that suggested in paper CoOp (Learning to Prompt for Vision-Language Models, Zhou et al, 2021 ) may be first utilised as a starting point before making adaptations to 1 query / class k-shots scenario to represent the loss function using the expression, max(Sn — Sp + M, 0)

[0083] Where S_n is the cosine similarity score of the negative example, S _p is the cosine similarity score of the positive example, and M>0 is a margin hyperparameter.

[0084] The Pairwise ranking loss (PWR) forces the text, positive image similarity to be higher than the text, negative image similarity by a margin M. Mathematically, when applied to S single query learning scenario, the PWR is a quadratic variation of triplet loss.

[0085] An approach to supplementing the calculated the loss function of the method according to this disclosure can be seen in Fig. 9, in which a Kullback-Leibler loss may be employed to supplement one or more of the above losses. The problem is that Learning with extremely few-shot samples may under-perform a zero-shot prediction (ZS). The idea is not to increase the distance from the zero-shot prediction too much in the wrong direction, by setting to zero any component of the gradient which conflicts with the gradient of the general knowledge, formally the gradient of the KL loss between the prediction of the zero-shot encoder and the learned encoder. The KL loss may be described mathematically as represented by the following expression,

[0086]

[0087] where pzs(wi|x) is the prediction of the original encoder, where Wi is 0 or 1 , and x is the sample image, p(t|x) is the prediction of the currently learned encoder, where ti is also 0 or 1. In this way, KL divergence may be used to modify the gradients of the “main” loss function value as described above.

[0088] In a further embodiment of this disclosure there is also provided a further method of differentiative control (Tiers Margin) of between correlations of tiers to query relevance, such as (but not limited to): a) hard positives (HP), b) hard-negatives (HN), and c) unrelated easy negatives (EN). Correlations may include: HP x HN, HP x EN, HN x EN. The margin coefficient Mi per correlation is a new type of hyperparameter introduced in this disclosure that is controllable (manually, or automatically) for the goal of optimizing the accuracy of the MM Prompt-Learning result. The sample pairings and their associated controllable margin coefficients can be seen in Fig. 10.

[0089] The concept of “Tiers" involves categorizing a group of items (samples) having similar level or degree of relevance to a particular query, denoted as N+2 (having 3 tiers or more). For instance, consider a scenario involving "uncovered manhole" images there could be these similar groups of images:

[0090] • Hard Positive (HP) (positive sample): Images depicting uncovered manholes, accurately representing high-relevance subjects with correct properties.

[0091] • Hard Negative (HN) (intermediate sample): Images showing covered manholes, correctly featuring the high- relevance subject (manhole) but with an incorrect low-relevance property.

[0092] • Easy Negative (EN) (negative sample): Images without manholes.

[0093] In standard contrastive learning, the “loss” is the difference between two items (item-1 to item-2) specifically chosen to be from different relevance groups. While standard learning typically uses a single margin coefficient M (representing punishment) over the loss of calculated distances between items from different groups (HPxHN, HPxEN, HNxEN), the Tiers Margin Mi of this disclosure assigns distinct controllable values. In other words, there is a method disclosed herein of determining query relevance in a multimodal system. The method comprising receiving a dataset relating to a query prompt, the dataset comprising one or more positive sample, one or more intermediate sample and one or more negative sample. The method may further comprise generating two or more pair of samples using the received dataset, each pair of samples comprised of either one positive sample and one negative sample, one positive sample and one intermediate sample, or one negative sample and one intermediate sample; and encoding each of the one or more pair of samples. The method may further comprise calculating a loss function for each encoded pair of samples; and determining a margin coefficient for each of the one or more pairs based on the calculated loss of each pair of samples; before adapting the query prompt based on the determined margin coefficient. It should be understood that the one or more intermediate sample may represent a negative sample that is relevant to the query prompt. The margin coefficient may represent a controllable value assigned to each pair of samples.

[0094] Importantly, this approach enables a differentiative control over different groups correlations, for example,

[0095] • Mi (Mi) (such as 1.0) in HP x HN items pairs (comparing query to items from tiers HP and HN);

[0096] • Mj (Mi) (such as 1.5) in HP x EN items pairs;

[0097] • Mk (Ma) (such as 0.5) in HN x EN items pairs.

[0098] This Tier Margin technique may lead, in some scenarios, to improved accuracy and higher generalization. The Tiers Margin Mi can be applied to any ranking learning mechanisms, including but not limited to those mentioned in the present disclosure of MM Retrieval Prompt-Learning, or other even classification DNN learning methods. The Mi values are hyperparameters, adjustable per query or class, which can be controlled manually by developers, or a by some sort of smart automation system. In the methods of this disclosure the query prompt is of a modality type that comprises text, images, audio, sensor data, LiDAR data, 3D visual data and / or video, logic data, transaction data, entity profiling data or state data. The state data may include data including user profile, or product or component multi part multi-modal definition or state).

[0099] The present disclosure also provides a system configured to perform multi-modal retrieval prompt learning. The system of this disclosure may comprise one or more processors configured to perform the methods described above. In some cases the methods described above may be performed by one or more neural networks as part of the system. Furthermore, there is provided a computer program stored in stored in non-transitory form comprising a program code for performing the methods disclosed herein when executed on a computer.

[0100] In addition, the technical solutions presented herein can used as part of additional systems and devices that have use of Al, DNN and CMMNN technologies for multimodal data processing, including model training and data-model inference. Data could be received from local sensors or received over network. For example, the present methods may be used as part of or receive data from the following:

[0101] • Robots and robotic devices

[0102] • Autonomous devices, such as drones and cars

[0103] • Wearable smart devices, such as smart glasses, watches

[0104] Personal computers and smartphones, such as those using operating systems such as Windows, MacOS, Linux, iOS, Android or other, that include multi-media files requiring semantic indexing and retrieval. Additionally, applications on these devices may include multimedia sensory data processing, such as video meetings software, browser software and more, in order to recognize with high accuracy multi-modal items of interest.

[0105] Some examples of devices that may be used to implement the method of the present disclosure may be electronic devices, computers, servers, network processing units, sensory systems (such as cameras), robotic devices and systems, user electronic devices (such as smartphones or wearable electronic devices), autonomous devices (such as cars or robots) etc. These devices may enable multi-modal data processing, retrieval and classification. The processing of the one or more processors described herein may be, for example only, on a smart device (such as smartphone, camera, robot or car), edge processing device or server; or, a remote processing means such as servers on private or public enterprise cloud.

[0106] Examples, of such devices, systems and / or software may use CMMNN models (such as CLIP, BLIP or any future multi-modal DNN based encoder) for the purpose of multimodal data processing, understanding, encoding (vector, storage in database systems supporting vectorized data types), for retrieval, as well as other cases (such as understanding and planning of actions). The system described herein may in some examples accept data items of modality A (such as image, video, audio, infrared and others) produced externally by sensory devices (such as cameras, microphones, radars) that could be deployed or part of more or less smart devices (such as smartphones, wearable devices, robots, CCTV cameras, drones, autonomous cars or other devices). The system may for example also accept or use from a pre-defined list, Query of modality B (such as text query), which may semantically or otherwise describe the required unlearnt “query” or “class”,

[0107] And may enable accuracy optimization per single arbitrary query, class or prompt, which can be used for retrieval (from a database), as well as other tasks (such as for classification).

[0108] The present disclosure therefore provides a method for MM Retrieval Prompt-Learning in Database Retrieval, including a method for learning 1 -Query to 1-Leamt-Prompt. There is also disclosed multiple adapted learning loss methods for MM Retrieval Prompt Learning for, some used as is and others with specific adaptations to support the hereby innovation. Finally, there is provided a method of differentiative control (Tiers Margin) of between correlations of tiers to query relevance, such as (but not limited to): a) hard positives (HP), b) hard-negatives (HN), and c) unrelated easy negatives (EN). Such Correlations include: HP x HN, HP x EN, and HN x EN. The margin coefficient per correlation (sample pairing) is a new type of hyperparameter, controllable (manually, or in the future automatically).

[0109] The present methods of this disclosure and therefore the systems disclosed herein that are configured to implement them provide several advantages such as ease of use, due to the low number of samples needed for learning, as opposed to model finetuning. Other advantages include no catastrophic forgetting, increased accuracy of retrieval given a learnt prompt, consistent accuracy and low computation cost as each query learnt independently. An additional advantage is that there is enhanced generalization (accuracy in different scenarios) and increased learning control due to the use of the margin coefficients (Tiered Margin Mi). The approaches disclosed herein also provide options for future research of automation of Mi values for accuracy optimization per scenarios. A further benefit of the methods and systems of this disclosure is the ease of use / low cost and low human effort due to low amount of samples needed for learning compared to model fine-tuning.

[0110] Definitions

[0111]

[0112]

Claims

CLAIMS1. A method for multi-modal retrieval prompt learning, the method comprising:receiving a dataset relating to a single query, the dataset comprising one or more positive sample and one or more negative sample;generating one or more pair of samples using the received dataset, each pair of samples comprised of one positive sample and one negative sample;encoding each of the one or more pair of samples;calculating a loss function for each encoded pair of samples based on a comparison of the encoded positive sample and the encoded negative sample to a learnt encoded query prompt;generating a learnt prompt for retrieval by optimising the calculated loss function; andoutputting the learnt prompt.

2. The method according to claim 1 , wherein the learnt query prompt represents the current learnt prompt state.

3. The method according to claim 1 or 2, wherein the comparison of the encoded positive sample and the encoded negative sample to the learnt encoded query prompt comprises calculating cosine similarity between each of the encoded positive sample and the encoded negative sample with the learnt encoded query prompt.

4. The method according to claim 3, wherein the method further comprises calculating the loss function using a model; andwherein the method further comprises updating the learnt encoded query prompt when the cosine similarity between the encoded negative sample and the learnt encoded query prompt is greater than the cosine similarity between the encoded positive sample and the learnt encoded query prompt.

5. The method according to any preceding claim, wherein the method further comprises the step of storing the output learnt prompt; andretrieving data items from an embeddings database using the learnt prompt.

6. The method according to any preceding claim, wherein the calculated loss function comprises calculating a marginized binary cross entropy loss function.

7. The method according to claim 6, wherein the marginized binary cross entropy loss function is calculated based on the expression:where y is the number between 1 and 0 representing the specific sample relevance, whether it’s value for a positive sample is 1, and its value for a negative sample is 0, and its value for the intermediate sample number between 1 and 0 which is controllable hyperparameter. Additionally, the p is an output number between 0 and 1.

8. The method according to any one of claims 1 to 5, wherein the calculated loss function comprises calculating a triplet loss function.

9. The method according to claim 8, wherein the triplet loss function is calculated based on the expression:max(Dp — Dn + M, 0)where Dp is the Euclidian distance between the (embedding of the) positive example and the anchor, Dn is the distance between the negative example and the anchor, and M>0 is a margin hyperparameter.

10. The method according to any one of claims 1 to 5, wherein the calculated loss function comprises calculating a pairwise ranking loss function.

11. The method according to any one of claims 10, wherein the pairwise ranking loss function is calculated based on the expression:max(Sn — Sp + M , 0)where Sn is the Cosine similarity score of the negative example, Sp is the cosine similarity score of the positive example, and M>0 is a margin hyperparameter.

12. The method according to any one of claims 1 to 5, wherein the calculated loss function is supplemented by applying a Kullback-Leibler loss function.

13. The method according to any one of claims 12, wherein the Kullback-Leibler loss function is calculated based on the expression:where pzs(wi|x) is the prediction of the original encoder, where Wi is 0 or 1 , and x is the sample image, p(ti|x) is the prediction of the currently learned encoder, where ti is also 0 or 1.

14. A method of determining query relevance in a multimodal system, the method comprising:receiving a dataset relating to a query prompt, the dataset comprising one or more positive sample, one or more intermediate sample and one or more negative sample;generating two or more pair of samples using the received dataset, each pair of samples comprised of either one positive sample and one negative sample, one positive sample and one intermediate sample, or one negative sample and one intermediate sample;encoding each of the one or more pair of samples;calculating a loss function for each encoded pair of samples;providing a margin coefficient for representing a weighting of the relevance of each of the one or more pairs; adapting the query prompt based on the provided margin coefficient.

15. The method according to claim 14, wherein the one or more intermediate sample represents a negative sample that is relevant to the query prompt.

16. The method according to any of claims 14 or 15, wherein the margin coefficient represents a controllable value assigned to each pair of samples; andwherein the margin coefficient is controlled via a hyper parameter.

17. The method according to any preceding claim, wherein query prompt is of a modality type that comprises text, images, audio, sensor data, LiDAR data, 3D visual data and / or video, logic data, transaction data, entity profiling data or state data.

18. A system configured to perform multi-modal retrieval prompt learning, the system comprising one or more processors configured to perform the method according to any preceding claim.

19. The method according to any preceding claim, wherein the method is performed by a neural network.

20. A computer program stored in stored in non-transitory form comprising a program code for performing the method according to any preceding claim when executed on a computer.

Images