Vector retrieval model training method, vector recall method, and electronic device

Abstract

Embodiments of the present disclosure disclose a vector retrieval model training method, a vector recall method, and an electronic device. The training method comprises: acquiring a training data set, the training data set comprising a plurality of data objects, a plurality of query conditions, and corresponding user behavior sequences; by using a pre-trained representation model, acquiring multimodal fused representation vectors of the query conditions and multimodal fused representation vectors of the data objects; generating semantically enhanced multimodal fused representation vectors corresponding to the data objects; inputting the multimodal fused representation vectors of the query conditions and the multimodal fused representation vectors of the data objects into a user tower in a vector retrieval model, and introducing, via the user behavior sequences, interaction between the multimodal fused representation vectors of the query conditions and the multimodal fused representation vectors of the data objects, so as to generate semantically enhanced multimodal fused representation vectors corresponding to a user. The embodiments of the present disclosure can improve model training efficiency and enhance the efficiency and relevance of personalized vector retrieval and recall results.

Description

Vector retrieval model training, vector recall methods and electronic devices

[0001] This disclosure claims priority to Chinese Patent Application No. 202510070219.7, filed with the China National Intellectual Property Administration on January 16, 2025, entitled “Vector Retrieval Model Training, Vector Recall Method and Electronic Device”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of personalized vector recall technology, and in particular to vector retrieval model training, vector recall methods, and electronic devices. Background Technology

[0003] Personalized vector recall is a technique used in recommender systems. Its core lies in using deep learning models to represent users and search objects as vectors. Specifically, the system maps users and search objects into a low-dimensional dense vector space. By calculating the similarity between user vectors and search object vectors, the system can find and recall search objects that best match the user's interests. The advantage of this method is that it can compute the search object vector offline, and then quickly find matching search objects online using efficient vector retrieval methods. Furthermore, it can incorporate more user features and contextual information to provide more personalized recommendations.

[0004] Personalized vector recall technology plays a crucial role in product search and retrieval within product information service systems (also known as e-commerce platforms), providing differentiated and efficient candidate products tailored to users' individual preferences. Initially, personalized vector recall primarily relied on vector representations of user and product textual features. However, recent studies have demonstrated that incorporating multimodal representations into personalized vector recall can improve the accuracy and semantic relevance of retrieved products. Therefore, personalized vector recall retrieval based on multimodal representations is a highly worthy research topic.

[0005] Existing technologies include personalized vector recall methods based on multimodal representation. Specifically, a multimodal feature extraction module is added to the personalized vector model to represent the product in both text and image formats, resulting in product text vectors and product image vectors. These two vectors are then fused to obtain the product's multimodal representation vector, which is then compared with the user's text representation vector to determine the matching product.

[0006] While the aforementioned approach can achieve personalized vector recall based on multimodal representation to some extent, it suffers from several drawbacks. First, it only provides multimodal representation for products, not for user features. Second, the personalized vector recall model employs a dual-tower structure: a "product tower" and a "user tower," used for vector representation of products and users respectively. Therefore, if multimodal representation of users is also required, an additional multimodal feature extraction module needs to be added to the "user tower," leading to a surge in the model's parameters and severely impacting performance. Furthermore, since the "product tower" and "user tower" are independent, even adding multimodal modules to each tower fails to achieve interaction between the multimodal representations of products and users, hindering the effective enhancement of efficiency and relevance metrics in personalized vector retrieval. Summary of the Invention

[0007] This disclosure provides vector retrieval model training, vector recall methods, and electronic devices that can improve model training efficiency and enhance the efficiency and relevance of personalized vector retrieval recalls.

[0008] This disclosure provides the following solutions:

[0009] A vector retrieval model training method, comprising:

[0010] Obtain a training dataset, which includes multiple data objects, multiple query conditions and corresponding user behavior sequences, wherein the user behavior sequence includes information about at least one data object related to the query conditions and associated with the user's operation behavior;

[0011] The multimodal fusion representation vectors of query conditions and data objects are obtained using pre-trained representation models.

[0012] By inputting the multimodal fusion representation vector of the data object into the data object tower of the dual-tower vector retrieval model, a semantically enhanced multimodal fusion representation vector corresponding to the data object is generated.

[0013] By inputting the query conditions and the multimodal fusion representation vector of the data object into the user tower of the vector retrieval model, and introducing the interaction between the multimodal fusion representation vector of the query conditions and the multimodal fusion representation vector of the data object through the user behavior sequence, a semantically enhanced multimodal fusion representation vector corresponding to the user is generated.

[0014] The similarity between the semantically enhanced multimodal fusion representation vector corresponding to the data object and the semantically enhanced multimodal fusion representation vector corresponding to the user is calculated, and multiple rounds of iterative calculation are performed according to a preset loss function to complete the training of the vector retrieval model.

[0015] The step of obtaining the multimodal fusion representation vector of the query conditions and the multimodal fusion representation vector of the data object using the pre-trained representation model includes:

[0016] The text representation vectors corresponding to the query conditions and data objects are obtained by using a pre-trained text representation model, and the image representation vectors corresponding to the query conditions and data objects are obtained by using a pre-trained image representation model.

[0017] The text representation vector of the query condition is fused with the image representation vector to obtain the multimodal fused representation vector of the query condition. The text representation vector of the data object is fused with the image representation vector to obtain the multimodal fused representation vector of the data object.

[0018] The semantically enhanced multimodal fusion representation vector corresponding to the generated data object includes:

[0019] The multimodal fusion representation vector of the data object is input into the data object tower in the vector retrieval model so as to fuse the multimodal fusion representation vector of the data object with the basic representation vector of the data object modeled by the vector retrieval model to obtain the semantically enhanced multimodal fusion representation vector corresponding to the data object.

[0020] The generation of the semantically enhanced multimodal fusion representation vector corresponding to the user includes:

[0021] The user multimodal behavior sequence is obtained based on the multimodal fusion representation vectors corresponding to the data objects included in the user behavior sequence.

[0022] By using an attention mechanism, the multimodal fusion representation vector of the query conditions corresponding to the user behavior sequence is fused with the multimodal fusion representation vector of the data objects in the user multimodal behavior sequence to obtain the user's multimodal fusion representation vector.

[0023] The multimodal fusion representation vector of the query conditions, the multimodal fusion representation vector of the user, and the basic representation vector of the user modeled by the vector retrieval model are fused together to obtain the semantically enhanced multimodal fusion representation vector of the user.

[0024] Among these, the process of fusing different representation vectors includes:

[0025] Different representation vectors are concatenated, and the high-dimensional representation vectors after concatenation are reduced in dimensionality using a multilayer perceptron structure to obtain a low-dimensional fused representation vector.

[0026] The step of performing multiple rounds of iterative calculations based on a preset loss function to train the vector retrieval model includes:

[0027] The similarity calculation result is used as the estimated click-through rate of the corresponding data object when the user inputs the query conditions. The vector retrieval model is trained using a contrastive learning loss function with the goal of maximizing the distance between the estimated click-through rates of positive and negative samples.

[0028] The positive and negative samples are determined based on the user's clicks on data objects in the training dataset.

[0029] A personalized vector recall method includes:

[0030] Upon receiving a search request submitted by a user, the system determines the query conditions associated with the search request and the user behavior sequence associated with the query conditions. The user behavior sequence includes at least one data object information associated with the user's operation behavior and related to the query conditions.

[0031] Obtain the multimodal fusion representation vector of the query conditions, and the multimodal representation vectors of multiple data objects in the database; the multimodal fusion representation vector of the query conditions and the multimodal representation vectors of the data objects are generated in advance based on a pre-trained representation model;

[0032] By inputting the multimodal fusion representation vector of the data object into the data object tower of the dual-tower vector retrieval model, a semantically enhanced multimodal fusion representation vector corresponding to the data object is generated.

[0033] By inputting the query conditions and the multimodal fusion representation vector of the data object into the user tower of the vector retrieval model, and introducing the interaction between the multimodal fusion representation vector of the query conditions and the multimodal fusion representation vector of the data object through the user behavior sequence, a semantically enhanced multimodal fusion representation vector corresponding to the user is generated.

[0034] The similarity between the semantically enhanced multimodal fusion representation vector corresponding to the data object and the semantically enhanced multimodal fusion representation vector corresponding to the user is calculated, and the personalized vector recall result is determined based on the similarity calculation result.

[0035] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of any of the preceding methods.

[0036] An electronic device, comprising:

[0037] One or more processors; and

[0038] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of any of the preceding methods.

[0039] A computer program product includes a computer program / computer executable instructions that, when executed by a processor in an electronic device, implement the steps of any of the preceding methods.

[0040] Based on the specific embodiments provided in this disclosure, the following technical effects are disclosed:

[0041] Through the embodiments of this disclosure, a pre-trained representation model can be used to achieve multimodal fusion representation of query conditions and data objects. This pre-obtained multimodal fusion representation can then be used as input information for a vector retrieval model to generate semantically enhanced multimodal fusion representation vectors for both data objects and users. Specifically, when generating semantically enhanced multimodal fusion representation vectors for users, the interaction between the multimodal fusion representation vectors of query conditions and data objects can be introduced through the user behavior sequence. In this way, since a pre-trained representation model is used to generate multimodal representation vectors, it is unnecessary to add a multimodal representation module to the vector retrieval model. Therefore, it avoids excessive parameter count in the vector retrieval model and fully utilizes the rich prior knowledge inherent in the pre-trained representation model. Furthermore, it enables interaction between the multimodal fusion representation vectors of query conditions and data objects. Finally, the similarity between the semantically enhanced multimodal fusion representation vectors generated for data objects and users can more accurately express the user's predicted click-through rate for specific data objects, thereby improving the model's training efficiency.

[0042] Of course, implementing any product of this disclosure does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 is a schematic diagram of the system architecture provided in an embodiment of this disclosure;

[0045] Figure 2 is a flowchart of the model training method provided in an embodiment of this disclosure;

[0046] Figure 3 is a flowchart of the personalized vector recall method provided in an embodiment of this disclosure;

[0047] Figure 4 is a schematic diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0048] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure are within the scope of protection of this disclosure.

[0049] To facilitate understanding of the specific implementation scheme provided in this disclosure, the concept of the "dual-tower" structure of the vector retrieval model is first briefly introduced below. Here, "tower" does not refer to a physical tower, but rather uses architectural terminology to vividly describe the model structure. The dual-tower structure refers to using two independent neural networks to encode users and data objects (e.g., items), respectively. This is suitable for retrieval tasks involving sparse vectors (such as high-dimensional feature vectors with most elements being zero), offering high retrieval efficiency and good scalability. Specifically, the User Tower is responsible for encoding user-side features to generate user vector representations; these features may include user attributes, historical behavior, etc. The Item Tower is responsible for encoding data object-side features to generate data object vector representations; these features may include data object attributes, text descriptions, tags, etc.

[0050] During the training phase, the dual-tower model learns effective user and data object representations by maximizing the inner product (or similarity metric) between the user vector and the corresponding positive sample data object vector, while minimizing the inner product between the user vector and the corresponding negative sample data object vector. In this way, users and data objects are mapped to the same latent feature space.

[0051] In the reasoning phase (i.e. the retrieval phase), given a user, its vector representation can be obtained through the user pyramid. Then, the similarity is calculated with all data object vectors (pre-calculated and stored by the data object pyramid) to quickly find the data objects most relevant to the user and complete the retrieval task.

[0052] Since the user tower and the data object tower are independent of each other, and in practical applications, only one pre-computation is often needed for the data object tower, this structure is conducive to efficient retrieval of large-scale data.

[0053] In this embodiment, an improvement is made to the implementation scheme of adding multimodal representation to the personalized vector retrieval model based on the aforementioned dual-tower structure. In the improved scheme, firstly, a pre-trained representation model can be used to obtain multimodal fusion representation vectors for both the Query (user-input keywords and other query conditions) and the Item (i.e., the object to be queried, for example, a product in a commodity information service system, or other types of data objects such as articles in other systems). Then, the multimodal fusion representation vectors of the Query and Item can be used as input information for the vector retrieval model to introduce multimodal representation.

[0054] In other words, in this embodiment, instead of directly adding a multimodal module to the vector retrieval model, a pre-trained representation model is used to obtain multimodal fusion representation vectors for the Query and Item, which are then input into the vector retrieval model as known information. This avoids making the vector retrieval model have an excessive number of parameters and also utilizes the rich prior knowledge inherent in the pre-trained representation model to achieve a more comprehensive and accurate multimodal representation vector for the Query and Item.

[0055] As shown in Figure 1, the pre-trained representation model can be divided into a pre-trained text representation model and a pre-trained image representation model, which are used to extract the text representation vectors of the query and the image representation vectors of the query and the item, respectively. Then, the text representation vector of the query and the image representation vector can be fused to obtain the multimodal fusion representation vector of the query, and the text representation vector of the item and the image representation vector can be fused to obtain the multimodal fusion representation vector of the item.

[0056] Furthermore, specifically in the process of introducing the multimodal fusion representation vectors of the Query and Item into the vector retrieval model, on the data object tower (I-tower) side, the multimodal fusion representation vector of the Item can be input into the I-tower of the vector retrieval model to generate the semantically enhanced multimodal fusion representation vector corresponding to the Item. On the user tower (U-tower) side, the multimodal fusion representation vectors of both the Query and Item can be input into the user tower, and the interaction between the multimodal fusion representation vectors of the Query and the multimodal fusion representation vectors of the Item can be introduced through user behavior sequences (which may include at least one Item information related to user operation behavior and the Query), thereby generating the semantically enhanced multimodal fusion representation vector corresponding to the user.

[0057] Finally, the similarity between the semantically enhanced multimodal fusion representation vector corresponding to the item and the semantically enhanced multimodal fusion representation vector corresponding to the user can be calculated, and the personalized vector recall result can be determined based on the similarity calculation result. Alternatively, during the training of the vector retrieval model, multiple rounds of iterative calculation can be performed according to a preset loss function to complete the training of the vector retrieval model.

[0058] By using the above method, the interaction between the multimodal fusion representation vector of the query and the multimodal fusion representation vector of the item can be achieved without making the number of parameters of the vector retrieval model too large. Furthermore, the rich prior knowledge of the pre-trained representation model itself can be fully utilized to obtain better multimodal fusion results.

[0059] The specific implementation schemes provided by the embodiments of this disclosure will be described in detail below.

[0060] Example 1

[0061] First, this embodiment provides a vector retrieval model training method, as shown in Figure 2. This method may include:

[0062] S201: Obtain a training data set, which includes multiple data objects, multiple query conditions and corresponding user behavior sequences, wherein the user behavior sequence includes information about at least one data object related to the query conditions and associated with the user's operation behavior.

[0063] In this training dataset, the data objects (Items) can be all data objects in the database, such as all products in a product information service system. The query conditions (Query) and user behavior sequences can be obtained from historical search records. The Query can be a large number of queries used by users during historical searches, including entered keywords. The user behavior sequence can include data objects related to a specific Query that the user has interacted with. For example, a user enters "down jacket" as a query to initiate a search. Prior to this, they may have clicked on and viewed some "down jacket" related products, added some "down jacket" related products to their favorites or shopping cart, etc. Furthermore, after initiating the search, they may have clicked on a product in the search results to view details, added a search result to their shopping cart, or even purchased a product. Therefore, the products associated with the actions before and after the search, especially those related to "down jacket," can constitute the user behavior sequence corresponding to the "down jacket" query. For example, a user behavior sequence can be (product 1, product 2, product 3, etc.). Of course, in the specific implementation, specific behavior type information can also be associated with specific products, such as click, add to cart, purchase, etc.

[0064] The actions performed before the search can be categorized into actions taken on the day of the search and historical actions taken within a recent period (e.g., 30 days). While different users may enter the same query criteria, the products associated with their actions before and after the search are usually different. Therefore, the same query can correspond to multiple user behavior sequences generated by different users.

[0065] S202: Use a pre-trained representation model to obtain the multimodal fusion representation vector of the query conditions and the multimodal fusion representation vector of the data object.

[0066] After obtaining the training dataset, the pre-trained representation model can be used to obtain the multimodal fusion representation vector of the query and the multimodal fusion representation vector of the item. The so-called multimodal fusion representation vector is a vector formed by fusing the representations of multiple modalities such as text and images for the same representation object.

[0067] The so-called pre-trained representation model is a model specifically designed for features extraction and other functions. Pre-trained models play a crucial role in natural language processing and image processing, enabling them to better understand and process large amounts of text and image data. Pre-trained models can help extract features and vector representations from text and image data, thereby improving the accuracy of downstream tasks. Specifically, pre-trained representation models can include pre-trained text representation models and pre-trained image representation models, used to extract text features and image features, respectively. Therefore, in a more specific implementation of this disclosure, the pre-trained text representation model can first be used to obtain the text representation vectors corresponding to the Query and Item, and the pre-trained image representation model can be used to obtain the image representation vectors corresponding to the Query and Item, respectively. Then, the text representation vector and image representation vector of the Query can be fused to obtain the multimodal fusion representation vector of the Query, and the text representation vector and image representation vector of the Item can be fused to obtain the multimodal fusion representation vector of the Item.

[0068] Typical pre-trained text representation models include BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-trained Transformer); pre-trained image representation models include BLIP (Bootstrapping Language-Image Pre-training) and CLIP (Contrastive Language-Image Pre-training), etc. This embodiment can utilize the above-mentioned pre-trained representation models to generate text representation vectors and image representation vectors.

[0069] Specifically, for a pre-trained text representation model, the titles of the query and item can be taken as input, and the mean of the token representation vectors of the last layer of the pre-trained text representation model can be taken as output to obtain the text vector representations of the query and item. For a pre-trained image representation model, assuming the BLIP model is used, the image of the item (e.g., a product) can be input into the BLIP image encoder to obtain the image representation vector of the item. For the query, since the query itself may be plain text content, the query can be input into BLIP's Q-Former (a lightweight Transformer model used to achieve efficient alignment between vision and language, using learnable query vectors to extract the most useful visual features for text generation from the frozen image encoder) to obtain the image representation vector of the query.

[0070] After obtaining the image representation vectors and text representation vectors of the Query and Item, the image representation vectors and text representation vectors can be fused to obtain a multimodal fused representation vector. To fully fuse the image representation vectors and text representation vectors and reduce memory usage during online prediction, this disclosure also provides a multimodal representation fusion model. This model includes two multimodal fusion dimensionality reduction structures for Query and Item. For the Query multimodal fusion dimensionality reduction structure, the previously extracted text representation vector and image representation vector of the Query can first be concatenated (for example, the "concat()" method can be used, which is used to concatenate two or more arrays without changing the existing arrays, only returning a copy of the concatenated array). Since the vector obtained after direct concatenation may be very long, for example, the text representation vector may be 768 dimensions and the image representation vector may be 256 dimensions, concatenating them together will result in a vector exceeding 1000 dimensions. Therefore, after concatenation, a structure such as MLP (Multilayer Perceptron) can be used for dimensionality reduction processing to finally obtain a low-dimensional multimodal fused vector representation, for example, 128 dimensions, etc., to reduce memory usage. Similarly, for an Item, the model can also concatenate the extracted text representation vector and image representation vector of the Item, and then use MLP to fuse and reduce the dimensionality, finally obtaining a low-dimensional multimodal fusion representation vector of the Item.

[0071] In other words, in this embodiment of the disclosure, instead of directly adding a multimodal representation module to the vector retrieval model, the text features and image features of the Query and Item are extracted in advance through pre-trained text representation models, image representation models, etc., and then further fused into a multimodal fusion representation vector of the Query and a multimodal fusion representation vector of the Item.

[0072] It should be noted that, in specific implementations, the multimodal fusion representation vectors of Query and Item can also be obtained through other means. For example, some pre-trained representation models have the ability to understand cross-modal interactions. Therefore, pre-trained multimodal models can be directly used to extract multimodal features from Query and Item.

[0073] After obtaining the multimodal fusion representation vector of the Query and the multimodal fusion representation vector of the Item, this information can be saved for subsequent vector retrieval calculations.

[0074] S203: By inputting the multimodal fusion representation vector of the data object into the data object tower of the vector retrieval model with a dual-tower structure, a semantically enhanced multimodal fusion representation vector corresponding to the data object is generated.

[0075] The aforementioned steps yield the multimodal fusion representation vectors of the Query and the Item, which can be saved. Subsequently, these multimodal fusion representation vectors can be used as input information for the vector retrieval model and participate in subsequent vector retrieval calculations.

[0076] Specifically, the personalized vector retrieval model is a type of recall model that can convert users, queries, and data objects into representation vectors, and then use these vectors to quickly compute...<U,Q,I> The vector dot product. For a given<U,Q> Personalized vector retrieval models select the top K items based on their internal integral ranking as the final search results. Currently, personalized vector retrieval models are widely used in search systems, providing personalized search results for each user. However, existing personalized vector retrieval models primarily focus on efficiency modeling, lacking modeling of semantic information such as text and images. Therefore, the recalled products are mainly high-efficiency, high-volume items (products whose definition, performance, quality, or price differs significantly from existing products, greatly exceeding user expectations and generating widespread word-of-mouth and multi-level marketing). However, for search systems, due to the limitations of the query terms, users' purchase intentions are relatively certain. Therefore, considering only the efficiency of the product while ignoring the relevance between the recalled product and the user's search query will severely impact the user experience and hinder transactions and conversions.

[0077] To improve the efficiency and relevance of personalized vector retrieval models, this disclosure proposes a personalized vector retrieval model based on multimodal fusion representation. This model can employ a dual-tower structure, consisting of a User tower and an Item tower. It calculates the inner product of the semantically enhanced multimodal user representation vector output from the User tower and the semantically enhanced multimodal item representation vector output from the Item tower, and then uses the resulting inner integral as a predicted CTR (Click-Through Rate).

[0078] In the Item Tower of the Personalized Vector Retrieval Model based on Multimodal Fusion Representation provided in this embodiment, the aforementioned multimodal fusion representation vector of the Item and the basic vector modeled by the personalized vector retrieval model for the Item can be further fused to obtain a semantically enhanced multimodal representation vector of the Item. That is, since the personalized vector retrieval model itself also has modeling capabilities and can model the Item to obtain a representation vector, this representation vector can be used as a basic representation vector and further fused with the aforementioned multimodal fusion representation vector of the Item to obtain richer information. Specifically, during the fusion, the multimodal fusion representation vector of the Item can be first concatenated with the aforementioned basic representation vector (using the concat() method), and then the connection result can be dimensionality reduced using an MLP algorithm structure to generate a low-dimensional (e.g., 128-dimensional) semantically enhanced multimodal representation vector for the Item.

[0079] S204: By inputting the query conditions and the multimodal fusion representation vector of the data object into the user tower of the vector retrieval model, and introducing the interaction between the multimodal fusion representation vector of the query conditions and the multimodal fusion representation vector of the data object through the user behavior sequence, a semantically enhanced multimodal fusion representation vector corresponding to the user is generated.

[0080] On the User side, the multimodal fusion representation vectors of both Query and Item can be used as input information. Furthermore, the interaction between the multimodal fusion representation vectors of Query and Item can be introduced by using user behavior sequences as a bridge, thereby generating semantically enhanced multimodal fusion representation vectors corresponding to the user.

[0081] In one specific implementation, the aforementioned multimodal fusion representation vector of the item can be combined with the user behavior sequence. For each item in the user behavior sequence, its multimodal fusion representation vector is obtained, thus yielding the user multimodal representation behavior sequence. That is, assuming a user behavior sequence is (item 1, item 2, item 3), the user multimodal representation behavior sequence can be (multimodal fusion representation vector of item 1, multimodal fusion representation vector of item 2, multimodal fusion representation vector of item 3).

[0082] Then, since the user behavior sequence is also related to the Query, the Query associated with the user behavior sequence can be determined. The Query multimodal fusion representation output from the previous steps and the user multimodal representation behavior sequence are fused based on the multimodal attention mechanism to express the semantic relevance between the Query and the specific Item in the sequence. The user multimodal behavior representation sequence is weighted and fused according to the semantic relevance to obtain the user multimodal vector.

[0083] Then, the query multimodal fusion representation vector, the user multimodal vector, and the basic user vector modeled by the personalized vector retrieval model can be further fused to generate an enhanced multimodal fusion representation vector for the user. Specifically, the aforementioned query multimodal fusion representation vector, user multimodal vector, and basic user vector can be concatenated, and then the concatenation result can be reduced using structures such as MLP to obtain a low-dimensional (e.g., 128-dimensional) enhanced multimodal fusion representation vector.

[0084] S205: Calculate the similarity between the semantically enhanced multimodal fusion representation vector corresponding to the data object and the semantically enhanced multimodal fusion representation vector corresponding to the user, and perform multiple rounds of iterative calculations based on a preset loss function to complete the training of the vector retrieval model.

[0085] Through the aforementioned steps S203 and S204, the semantically enhanced multimodal fusion representation vector corresponding to the item and the semantically enhanced multimodal fusion representation vector corresponding to the user are obtained. Then, similarity calculation can be performed between the two, for example, by calculating the inner product. The result can be used as a click-through rate (CTR) estimate for the corresponding item when the user inputs the corresponding query. The training of the vector retrieval model is completed by optimizing the accuracy of this CTR estimate. For example, it can be used...<U,Q,I> All click samples (if a user clicks on an item while entering a query to search, then that item...)<U,Q,I> The model uses the clicked sample as a positive sample and performs negative sampling within a batch (which can be a subset of the training dataset). All other samples in the batch except the current clicked sample are used as negative samples. Then, a contrastive learning loss function (such as Hinge Loss) is used to maximize the distance between the positive and negative samples to train the vector retrieval model.

[0086] In summary, through the embodiments of this disclosure, a pre-trained representation model can be used to achieve multimodal fusion representation of query conditions and data objects. This pre-obtained multimodal fusion representation can then be used as input information for a vector retrieval model to generate semantically enhanced multimodal fusion representation vectors for both data objects and users. Specifically, when generating semantically enhanced multimodal fusion representation vectors for users, the interaction between the multimodal fusion representation vectors of query conditions and data objects can be introduced through the user behavior sequence. In this way, since a pre-trained representation model is used to generate multimodal representation vectors, it is unnecessary to add a multimodal representation module to the vector retrieval model. Therefore, the number of parameters in the vector retrieval model can be avoided, and the rich prior knowledge inherent in the pre-trained representation model can be fully utilized. Furthermore, the interaction between the multimodal fusion representation vectors of query conditions and data objects can be realized. Finally, the similarity between the semantically enhanced multimodal fusion representation vectors generated for data objects and users can more accurately express the user's predicted click-through rate for specific data objects, thereby improving the training efficiency of the model.

[0087] Example 2

[0088] The aforementioned Example 1 mainly introduced the training method of the vector retrieval model. In this Example 2, when using the vector retrieval model for vector recall, the multimodal fusion representation vectors of query conditions and data objects can also be obtained in advance using a pre-trained representation model and saved. When a user inputs a specific query to initiate a search, the aforementioned multimodal fusion representation vectors can be input into the vector retrieval model to obtain recall results. Since the rich prior knowledge inherent in the pre-trained representation model can be fully utilized, and the interaction between the multimodal fusion representation vectors of query conditions and the multimodal fusion representation vectors of data objects can be realized, the relevance between the recall results and the query conditions can be improved to a certain extent.

[0089] Specifically, this second embodiment provides a personalized vector recall method, as shown in Figure 3. This method may specifically include:

[0090] S301: After receiving a search request submitted by a user, determine the query conditions associated with the search request and the user behavior sequence associated with the query conditions, wherein the user behavior sequence includes at least one data object information associated with the query conditions related to the user operation behavior.

[0091] This second embodiment describes the specific process of the inference phase. The inference result refers to the process of using the vector retrieval model to perform specific vector retrieval after training the model. In this process, the user will input a specific query to initiate a search request. Therefore, in a single request, the specific user, the query, and the corresponding user behavior sequence can all be fixed. The user behavior sequence mainly consists of items related to the current query from historical actions performed by the user before initiating the search request or from items related to the current action (primarily actions performed on the same day as the current search).

[0092] S302: Obtain the multimodal fusion representation vector of the query conditions and the multimodal representation vectors of multiple data objects in the database; the multimodal fusion representation vector of the query conditions and the multimodal representation vectors of the data objects are generated in advance based on a pre-trained representation model.

[0093] The multimodal fusion representation vector of the query can also be pre-generated using a pre-trained representation model and stored in a database. This way, after a user initiates a specific search request, the multimodal fusion representation vector of the currently input query can be obtained by querying the database. Upon receiving the specific query input from the user, the system can first determine if the query exists in the database. If it does, the corresponding multimodal fusion representation vector can be directly retrieved. If the query does not exist in the database, multiple queries similar to the current query can be retrieved, and the average of their multimodal fusion representation vectors can be used as the multimodal fusion representation vector of the current query, and so on.

[0094] S303: By inputting the multimodal fusion representation vector of the data object into the data object tower of the vector retrieval model with a dual-tower structure, a semantically enhanced multimodal fusion representation vector corresponding to the data object is generated.

[0095] S304: By inputting the query conditions and the multimodal fusion representation vector of the data object into the user tower of the vector retrieval model, and introducing the interaction between the multimodal fusion representation vector of the query conditions and the multimodal fusion representation vector of the data object through the user behavior sequence, a semantically enhanced multimodal fusion representation vector corresponding to the user is generated.

[0096] Steps S303 and S304 are basically the same as steps S203 and S204, and will not be described in detail here.

[0097] S305: Calculate the similarity between the semantically enhanced multimodal fusion representation vector corresponding to the data object and the semantically enhanced multimodal fusion representation vector corresponding to the user, and determine the personalized vector recall result based on the similarity calculation result.

[0098] After obtaining the semantically enhanced multimodal fusion representation vectors corresponding to the item and the user, similarity can be calculated between them. Since there may be multiple items, the semantically enhanced multimodal fusion representation vector corresponding to the user can be compared with the semantically enhanced multimodal fusion representation vectors corresponding to each item. The resulting similarity can represent the probability predicted by the model that a user will click on a corresponding item in the current query state. This result can then be used for vector recall. For example, items with similarity results higher than a certain threshold, or those ranked in the top N, can be returned to the user as recall results, and so on.

[0099] In this way, the rich prior knowledge of the pre-trained representation model can be fully utilized, and the interaction between the multimodal fusion representation vector of the query conditions and the multimodal fusion representation vector of the data object can be realized. Therefore, the relevance and quality of the recall results can be improved to a certain extent, thereby enhancing the user experience.

[0100] For the parts of this embodiment that are not described in detail, please refer to the description in embodiment one and other parts of this specification, which will not be repeated here.

[0101] It should be noted that the embodiments disclosed herein may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country in which it is located (e.g., with the user's explicit consent, with the user being properly notified, etc.).

[0102] Corresponding to Embodiment 1, this disclosure also provides a vector retrieval model training apparatus, which may include:

[0103] The training dataset acquisition unit is used to acquire a training dataset, which includes multiple data objects, multiple query conditions and corresponding user behavior sequences. The user behavior sequence includes information about at least one data object related to the query conditions and associated with the user's operation behavior.

[0104] The multimodal representation unit is used to obtain the multimodal fusion representation vector of the query conditions and the multimodal fusion representation vector of the data object using a pre-trained representation model.

[0105] The first semantic enhancement unit is used to generate a semantically enhanced multimodal fusion representation vector corresponding to the data object by inputting the multimodal fusion representation vector of the data object into the data object tower in the vector retrieval model with a dual-tower structure.

[0106] The second semantic enhancement unit is used to input the query conditions and the multimodal fusion representation vector of the data object into the user tower of the vector retrieval model, and introduce the interaction between the multimodal fusion representation vector of the query conditions and the multimodal fusion representation vector of the data object through the user behavior sequence, so as to generate the semantically enhanced multimodal fusion representation vector corresponding to the user.

[0107] The similarity calculation unit is used to calculate the similarity between the semantically enhanced multimodal fusion representation vector corresponding to the data object and the semantically enhanced multimodal fusion representation vector corresponding to the user, and to perform multiple rounds of iterative calculation according to a preset loss function to complete the training of the vector retrieval model.

[0108] Specifically, the multimodal characterization unit can be used for:

[0109] The text representation vectors corresponding to the query conditions and data objects are obtained by using a pre-trained text representation model, and the image representation vectors corresponding to the query conditions and data objects are obtained by using a pre-trained image representation model.

[0110] The text representation vector of the query condition is fused with the image representation vector to obtain the multimodal fused representation vector of the query condition. The text representation vector of the data object is fused with the image representation vector to obtain the multimodal fused representation vector of the data object.

[0111] The first semantic enhancement unit can be specifically used to: input the multimodal fusion representation vector of the data object into the data object tower in the vector retrieval model, so as to fuse the multimodal fusion representation vector of the data object with the basic representation vector of the data object modeled by the vector retrieval model, and obtain the semantically enhanced multimodal fusion representation vector corresponding to the data object.

[0112] The second semantic enhancement unit can be specifically used to: obtain a user multimodal behavior sequence based on the multimodal fusion representation vectors corresponding to the data objects included in the user behavior sequence;

[0113] By using an attention mechanism, the multimodal fusion representation vector of the query conditions corresponding to the user behavior sequence is fused with the multimodal fusion representation vector of the data objects in the user multimodal behavior sequence to obtain the user's multimodal fusion representation vector.

[0114] The multimodal fusion representation vector of the query conditions, the multimodal fusion representation vector of the user, and the basic representation vector of the user modeled by the vector retrieval model are fused together to obtain the semantically enhanced multimodal fusion representation vector of the user.

[0115] In the process of fusing different representation vectors, the different representation vectors can be connected, and the high-dimensional representation vector after connection can be reduced in dimensionality using a multilayer perceptron structure to obtain a low-dimensional fused representation vector.

[0116] Specifically, during training, the result of the similarity calculation can be used as the estimated value of the click-through rate that the corresponding data object can obtain when the user inputs the query conditions. The vector retrieval model is trained by using the contrastive learning loss function to maximize the distance between the estimated click-through rates of positive and negative samples.

[0117] The positive and negative samples are determined based on the user's clicks on data objects in the training dataset.

[0118] Corresponding to Embodiment 2, this disclosure also provides a personalized vector recall device, which may include:

[0119] The request receiving unit is configured to, upon receiving a search request submitted by a user, determine the query conditions associated with the search request and the user behavior sequence associated with the query conditions, wherein the user behavior sequence includes at least one data object information associated with the query conditions and related to the user operation behavior.

[0120] A multimodal vector acquisition unit is used to acquire the multimodal fusion representation vector of the query conditions and the multimodal representation vectors of multiple data objects in the database; the multimodal fusion representation vector of the query conditions and the multimodal representation vectors of the data objects are generated in advance based on a pre-trained representation model;

[0121] The first semantic enhancement unit is used to generate a semantically enhanced multimodal fusion representation vector corresponding to the data object by inputting the multimodal fusion representation vector of the data object into the data object tower in the vector retrieval model with a dual-tower structure.

[0122] The second semantic enhancement unit is used to input the query conditions and the multimodal fusion representation vector of the data object into the user tower of the vector retrieval model, and introduce the interaction between the multimodal fusion representation vector of the query conditions and the multimodal fusion representation vector of the data object through the user behavior sequence, so as to generate the semantically enhanced multimodal fusion representation vector corresponding to the user.

[0123] The similarity calculation unit is used to calculate the similarity between the semantically enhanced multimodal fusion representation vector corresponding to the data object and the semantically enhanced multimodal fusion representation vector corresponding to the user, and to determine the personalized vector recall result based on the similarity calculation result.

[0124] In addition, this disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0125] And an electronic device, comprising:

[0126] One or more processors; and

[0127] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0128] A computer program product includes a computer program / computer executable instructions that, when executed by a processor in an electronic device, implement the steps of the method described in the foregoing method embodiments.

[0129] Figure 4 illustrates the architecture of an electronic device, which may include a processor 410, a video display adapter 411, a disk drive 412, an input / output interface 413, a network interface 414, and a memory 420. The processor 410, video display adapter 411, disk drive 412, input / output interface 413, network interface 414, and memory 420 can communicate with each other via a communication bus 430.

[0130] The processor 410 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in this disclosure.

[0131] The memory 420 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 420 can store the operating system 421 for controlling the operation of the electronic device 400, and the basic input / output system (BIOS) for controlling the low-level operations of the electronic device 400. Additionally, it can store a web browser 423, a data storage management system 424, and a vector retrieval processing system 425, etc. The aforementioned vector retrieval processing system 425 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when the technical solution provided by this disclosure is implemented through software or firmware, the relevant program code is stored in the memory 420 and is called and executed by the processor 410.

[0132] Input / output interface 413 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0133] Network interface 414 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0134] Bus 430 includes a pathway for transmitting information between various components of the device, such as processor 410, video display adapter 411, disk drive 412, input / output interface 413, network interface 414, and memory 420.

[0135] It should be noted that although the above-described device only shows the processor 410, video display adapter 411, disk drive 412, input / output interface 413, network interface 414, memory 420, bus 430, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the present disclosure, and not necessarily all the components shown in the figures.

[0136] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this disclosure can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this disclosure.

[0137] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0138] The foregoing has provided a detailed description of the vector retrieval model training, vector recall method, and electronic device provided in this disclosure. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the embodiments above are merely for the purpose of helping to understand the methods and core ideas of this disclosure. Furthermore, those skilled in the art will recognize that, based on the ideas of this disclosure, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. A method for training a vector retrieval model, wherein, include: Obtain a training dataset, which includes multiple data objects, multiple query conditions and corresponding user behavior sequences, wherein the user behavior sequence includes information about at least one data object related to the query conditions and associated with the user's operation behavior; The multimodal fusion representation vectors of query conditions and data objects are obtained using pre-trained representation models. By inputting the multimodal fusion representation vector of the data object into the data object tower of the dual-tower vector retrieval model, a semantically enhanced multimodal fusion representation vector corresponding to the data object is generated. By inputting the query conditions and the multimodal fusion representation vector of the data object into the user tower of the vector retrieval model, and introducing the interaction between the multimodal fusion representation vector of the query conditions and the multimodal fusion representation vector of the data object through the user behavior sequence, a semantically enhanced multimodal fusion representation vector corresponding to the user is generated. The similarity between the semantically enhanced multimodal fusion representation vector corresponding to the data object and the semantically enhanced multimodal fusion representation vector corresponding to the user is calculated, and multiple rounds of iterative calculation are performed according to a preset loss function to complete the training of the vector retrieval model.

2. The method according to claim 1, wherein, The process of obtaining the multimodal fusion representation vector of the query conditions and the multimodal fusion representation vector of the data object using the pre-trained representation model includes: The text representation vectors corresponding to the query conditions and data objects are obtained by using a pre-trained text representation model, and the image representation vectors corresponding to the query conditions and data objects are obtained by using a pre-trained image representation model. The text representation vector of the query condition is fused with the image representation vector to obtain the multimodal fused representation vector of the query condition. The text representation vector of the data object is fused with the image representation vector to obtain the multimodal fused representation vector of the data object.

3. The method according to claim 2, wherein, The pre-trained text representation model includes BERT or GPT, and the pre-trained image representation model includes BLIP or CLIP, used to extract text representation vectors and image representation vectors of query conditions and data objects.

4. The method according to any one of claims 1-3, wherein, The semantically enhanced multimodal fusion representation vector corresponding to the generated data object includes: The multimodal fusion representation vector of the data object is input into the data object tower in the vector retrieval model so as to fuse the multimodal fusion representation vector of the data object with the basic representation vector of the data object modeled by the vector retrieval model to obtain the semantically enhanced multimodal fusion representation vector corresponding to the data object.

5. The method according to claim 4, wherein, The basic representation vector of the data object is modeled using a personalized vector retrieval model, representing the basic feature information of the data object in the search results.

6. The method according to any one of claims 1-5, wherein, The generated semantically enhanced multimodal fusion representation vector corresponding to the user includes: The user multimodal behavior sequence is obtained based on the multimodal fusion representation vectors corresponding to the data objects included in the user behavior sequence. By using an attention mechanism, the multimodal fusion representation vector of the query conditions corresponding to the user behavior sequence is fused with the multimodal fusion representation vector of the data objects in the user multimodal behavior sequence to obtain the user's multimodal fusion representation vector. The multimodal fusion representation vector of the query conditions, the multimodal fusion representation vector of the user, and the basic representation vector of the user modeled by the vector retrieval model are fused together to obtain the semantically enhanced multimodal fusion representation vector of the user.

7. The method according to claim 6, wherein, An attention mechanism is used to weight the multimodal fusion representation vector of each data object in the user's multimodal behavior sequence, and the fusion process is performed based on its relevance to the query conditions.

8. The method according to any one of claims 2 to 7, wherein, When fusing different representation vectors, the following steps are included: Different representation vectors are concatenated, and the high-dimensional representation vectors after concatenation are reduced in dimensionality using a multilayer perceptron structure to obtain a low-dimensional fused representation vector.

9. The method according to claim 8, wherein, The multilayer perceptron structure includes at least one hidden layer for performing nonlinear transformation and dimensionality reduction on the connected high-dimensional representation vector.

10. The method according to any one of claims 2 to 9, wherein, The step of performing multiple rounds of iterative calculations based on a preset loss function to complete the training of the vector retrieval model includes: The similarity calculation result is used as the estimated click-through rate of the corresponding data object when the user inputs the query conditions. The vector retrieval model is trained using a contrastive learning loss function with the goal of maximizing the distance between the estimated click-through rates of positive and negative samples. The positive and negative samples are determined based on the user's clicks on data objects in the training dataset.

11. The method according to claim 10, wherein, The contrastive learning loss function includes Hinge Loss or Triplet Loss, which is used to optimize the distance between positive and negative samples and improve the accuracy of similarity calculation.

12. The method according to any one of claims 1 to 11, wherein, When obtaining the multimodal fusion representation vector of the query conditions, the query conditions are input into the Q-Former module in the pre-trained image representation model to generate the image representation vector of the query conditions.

13. The method according to any one of claims 1 to 12, wherein, The user behavior sequence further includes the specific operation types performed by the user on the data object, including but not limited to clicking, adding to favorites, and purchasing, to enhance the correlation between the user behavior sequence and the query conditions.

14. The method according to any one of claims 1 to 13, wherein, The dual-tower vector retrieval model includes a user tower and a data object tower, which are used to process query conditions and user behavior sequences, as well as the multimodal fusion representation vectors of data objects, respectively.

15. The method according to any one of claims 1 to 14, wherein, After training the vector retrieval model, the model performance is verified using a test dataset, and the model parameters are adjusted based on the verification results to optimize the recall effect.

16. The method according to any one of claims 1 to 15, wherein, The similarity calculation is not limited to inner product calculation; cosine similarity or other mathematical methods suitable for measuring the similarity between vectors can also be used.

17. A personalized vector recall method, wherein, include: Upon receiving a search request submitted by a user, the system determines the query conditions associated with the search request and the user behavior sequence associated with the query conditions. The user behavior sequence includes at least one data object information associated with the user's operation behavior and related to the query conditions. Obtain the multimodal fusion representation vector of the query conditions, and the multimodal representation vectors of multiple data objects in the database; the multimodal fusion representation vector of the query conditions and the multimodal representation vectors of the data objects are generated in advance based on a pre-trained representation model; By inputting the multimodal fusion representation vector of the data object into the data object tower of the dual-tower vector retrieval model, a semantically enhanced multimodal fusion representation vector corresponding to the data object is generated. By inputting the query conditions and the multimodal fusion representation vector of the data object into the user tower of the vector retrieval model, and introducing the interaction between the multimodal fusion representation vector of the query conditions and the multimodal fusion representation vector of the data object through the user behavior sequence, a semantically enhanced multimodal fusion representation vector corresponding to the user is generated. The similarity between the semantically enhanced multimodal fusion representation vector corresponding to the data object and the semantically enhanced multimodal fusion representation vector corresponding to the user is calculated, and the personalized vector recall result is determined based on the similarity calculation result.

18. A computer-readable storage medium having a computer program stored thereon, wherein, When executed by a processor, the program performs the steps of the method described in any one of claims 1 to 16.

19. An electronic device, wherein, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 16.

20. A computer program product comprising a computer program / computer-executable instructions, wherein, When the computer program / computer-executable instructions are executed by a processor in an electronic device, they implement the steps of the method according to any one of claims 1 to 16.

Images