Model processing method and apparatus, computer program product, and device

Abstract

The present application discloses a model processing method and apparatus, a computer program product, and a device. The method comprises: acquiring content data of a multimedia resource, and interaction data between a sample object and the multimedia resource; using the content data and the interaction data to perform adaptive training on a recommendation model to obtain a first recommendation model; acquiring a collaborative feature, the collaborative feature including at least one of features extracted from the interaction data and representing correlations between objects, between objects and resources, and between resources; and using the collaborative feature to perform optimization training on the first recommendation model to obtain a trained recommendation model, the trained recommendation model being used for determining a multimedia resource recommended to an object.

Description

Model processing methods, model processing devices, computer program products and equipment

[0001] This application claims priority to Chinese Patent Application No. 202411700655.X, filed on November 25, 2024, entitled “Model Processing Method, Model Processing Apparatus, Computer Program Product and Equipment”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence, and in particular to a model processing method, a model processing device, a computer program product, and an equipment.

[0003] Background of the Invention

[0004] In the scenario of recommending multimedia resources, a trained model can be used to generate features of multimedia resources and features of the user to be recommended resources. Thus, multimedia resources can be recommended to the user based on the similarity between the features of the multimedia resources and the features of the user. For example, multimedia resources with features similar to the user's features can be recommended to the user. Summary of the Invention

[0005] This application provides a model processing method, a model processing apparatus, a computer program product, and a device that can improve the accuracy of recommending multimedia resources to objects.

[0006] This application provides a model processing method, the method comprising:

[0007] Acquire the content data of multimedia resources in the multimedia resource set, and the interaction data of sample objects with respect to the multimedia resources in the multimedia resource set;

[0008] The recommendation model is adaptively trained using content data and interaction data to obtain the first recommendation model;

[0009] Acquire collaborative features; collaborative features include at least one of the features of correlation between objects, between objects and resources, and between resources extracted from interaction data.

[0010] The first recommendation model is optimized and trained using collaborative features to obtain the trained recommendation model; the trained recommendation model is used to determine the multimedia resources to recommend to the target.

[0011] This application provides a model processing apparatus, the apparatus comprising:

[0012] The first acquisition module is used to acquire the content data of multimedia resources in the multimedia resource set and the interaction data of sample objects with respect to the multimedia resources in the multimedia resource set.

[0013] The training module is used to adaptively train the recommendation model using content data and interaction data to obtain the first recommendation model;

[0014] The second acquisition module is used to acquire collaborative features; the collaborative features include at least one of the features of correlation between objects, between objects and resources, and between resources extracted from the interaction data.

[0015] The optimization module is used to optimize and train the first recommendation model using collaborative features to obtain the trained recommendation model; the trained recommendation model is used to determine the multimedia resources to recommend to the target.

[0016] This application provides a computer device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the method of this application.

[0017] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the method described in the above-mentioned aspect.

[0018] According to one aspect of this application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the methods of various alternative embodiments, including the one described above.

[0019] The method proposed in this application allows for multi-stage progressive training of the recommendation model. In this multi-stage progressive training, the recommendation model undergoes initial overall adaptive training, enabling it to learn information related to multimedia resources and objects. Then, collaborative features (such as multimedia collaborative features of multimedia resources and / or object collaborative features of sample objects) are introduced to further optimize the training of the recommendation model, allowing it to learn the existing implicit features of multimedia resources and / or sample objects in online recommendation scenarios. This significantly improves the training effect of the recommendation model, resulting in a superior model that can accurately recall multimedia resources recommended to objects.

[0020] Brief description of the attached figures

[0021] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 is a schematic diagram of the network architecture of a recommendation network according to an embodiment of this application;

[0023] Figure 2 is a schematic diagram of a model training scenario according to an embodiment of this application;

[0024] Figure 3 is a flowchart illustrating a model processing method according to an embodiment of this application;

[0025] Figure 4 is a schematic diagram of the effect of a recommended interface for displaying multimedia resources according to an embodiment of this application;

[0026] Figure 5 is a schematic diagram of a process for optimizing and training a first recommendation model according to an embodiment of this application;

[0027] Figure 6 is a schematic diagram of the structure of a first attention mask according to an embodiment of this application;

[0028] Figure 7 is a schematic diagram of a scenario for optimizing and training a first recommendation model according to an embodiment of this application;

[0029] Figure 8 is a schematic diagram of a process for optimizing and training a second recommendation model according to an embodiment of this application;

[0030] Figure 9 is a schematic diagram of the structure of a second attention mask according to an embodiment of this application;

[0031] Figure 10 is a schematic diagram of a scenario for optimizing and training a second recommendation model according to an embodiment of this application;

[0032] Figure 11 is a schematic diagram of a process for recommending multimedia resources to an object according to an embodiment of this application;

[0033] Figure 12 is a schematic diagram of a scenario in which multimedia resources are recommended to a user according to an embodiment of this application;

[0034] Figure 13 is a schematic diagram of a model processing device according to an embodiment of this application;

[0035] Figure 14 is a schematic diagram of the structure of a computer device according to an embodiment of this application.

[0036] Methods of implementing the present invention

[0037] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] All data collected in this application (such as content data of multimedia resources, interaction data of sample objects with respect to multimedia resources, multimedia collaborative features, and related data such as object collaborative features) are collected with the consent and authorization of the data subject (such as users, institutions, or enterprises), and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant regions.

[0039] Artificial intelligence (AI) is a new technical science that studies, develops, and applies theories, methods, technologies, and application systems to simulate, extend, and expand human intelligence. This application primarily concerns machine learning within artificial intelligence.

[0040] LLM: Large Language Model, a natural language processing model based on deep learning, designed to understand and generate natural language text. LLMs typically consist of hundreds of millions to hundreds of billions of parameters and are capable of handling complex language tasks.

[0041] Embedding: A vector representation of data that has been embedded.

[0042] Prefix-tuning: An efficient parameter fine-tuning method designed to adapt to a specific task by adding a trainable "prefix" to the input sequence without adjusting all parameters of the entire model.

[0043] GNN: A deep learning model for processing and analyzing graph-structured data, which learns and propagates information by utilizing the relationships between nodes.

[0044] Figure 1 is a schematic diagram of the network architecture of a recommended network according to an embodiment of this application. As shown in Figure 1, the network architecture may include a server 200 and a cluster of terminal devices. The cluster of terminal devices may include at least one terminal device, and the number of terminal devices is not limited here. As shown in Figure 1, the multiple terminal devices may specifically include terminal device 1, terminal device 2, terminal device 3, ..., terminal device n. Terminal device 1, terminal device 2, terminal device 3, ..., terminal device n can all connect to the server 200 via the network, thereby interacting with the server 200 through the network connection.

[0045] As shown in Figure 1, server 200 can be a standalone physical server, a server cluster or distributed system consisting of at least two physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Terminal devices can be: smartphones, tablets, laptops, desktop computers, smart TVs, in-vehicle terminals, smart home devices, and other smart terminals. The following description uses the communication between terminal device 1 and server 200 as an example.

[0046] In this system, terminal device 1 can be a user's terminal. Terminal device 1 may include a client that can receive and view multimedia resources recommended to the user. Server 200 can be a backend server for the client. Server 200 can recall (i.e., determine or filter) the multimedia resources recommended to the user and push these resources to the client, allowing the client to display the received multimedia resources on its front-end interface for the user to view. Server 200 can recall the multimedia resources recommended to the user using a trained recommendation model. The overall training process of this model can be found in the description of the corresponding embodiment in Figure 2, and the process of recalling the recommended multimedia resources using this model can be found in the relevant description of the corresponding embodiment in Figure 11.

[0047] Figure 2 is a schematic diagram of a model training scenario according to an embodiment of this application. As shown in Figure 2, the server 200 in Figure 1 can obtain a multimedia resource set, which may include at least one multimedia resource for training the recommendation model. The server 200 can obtain the content data of the multimedia resources in the multimedia resource set, as well as the interaction data of the sample object with respect to the multimedia resources in the multimedia resource set. The content data reflects the specific content of the multimedia resources, and the interaction data reflects the interaction behavior of the sample object with respect to the multimedia resources, such as click behavior.

[0048] In addition, server 200 can also acquire collaborative features. These collaborative features may include at least one of the following: relevance features related to multimedia resources in the multimedia resource set (referred to as multimedia collaborative features), and relevance features related to sample objects (referred to as object collaborative features). The multimedia collaborative features may be features learned and generated from multimedia resources in an online recommendation scenario. The object collaborative features may be features learned and generated from sample objects in an online recommendation scenario. The online recommendation scenario where the multimedia resources reside and the online recommendation scenario where the sample objects reside may be different or the same. The following example, where the collaborative features simultaneously include both multimedia collaborative features and object collaborative features, will illustrate how to train the recommendation model.

[0049] The server can first use the aforementioned content data and interaction data to adaptively train the recommendation model (also known as pre-training) to obtain a first recommendation model. This first recommendation model can initially learn the content features of multimedia resources and the preference features of sample objects for multimedia resources. The server 200 can incorporate (i.e., adopt) multimedia collaborative features to optimize the training of the first recommendation model to obtain a second recommendation model. This second recommendation model can effectively incorporate multimedia collaborative features of multimedia resources. The server 200 can also incorporate object collaborative features to further optimize the training of the second recommendation model to obtain a final trained recommendation model. This final trained recommendation model can then effectively incorporate object collaborative features of sample objects.

[0050] Using the method of this application, the recommendation model can be progressively trained through the pipeline described above, gradually injecting multimedia collaborative information and object collaborative information into the recommendation model. This allows the recommendation model to not only learn the content features of the multimedia resources themselves and the preference features of the sample objects for the multimedia resources, but also to learn the rich features (such as multimedia collaborative features and object collaborative features) that the multimedia resources and sample objects already have in other sources (such as online recommendation scenarios). As a result, a recommendation model with excellent training performance can be obtained. Through this trained recommendation model, the multimedia resources recommended to the object can be accurately recalled.

[0051] Figure 3 is a flowchart illustrating a model processing method according to an embodiment of this application. The executing entity in this embodiment can be a model processing device (hereinafter referred to as a processing device). This processing device can be a single computer device or a cluster of multiple computer devices. The computer device can be a server or other devices; this application does not impose any limitations on this. As shown in Figure 3, the method may include the following steps.

[0052] Step S101: Obtain the content data of the multimedia resources in the multimedia resource set and the interaction data of the sample objects with respect to the multimedia resources in the multimedia resource set.

[0053] For example, the processing device can acquire content data of multimedia resources from a multimedia resource set, which can be a collection of multimedia resources used to train a recommendation model. The content data of the multimedia resources can be data used to refer to (or describe) the resource content of the multimedia resources. These multimedia resources can be any type of resource to be recommended, such as videos, images, articles, products, or live streams.

[0054] In this application, the content data of a multimedia resource can be text data (i.e., data in text format). If the multimedia resource itself is a text-type resource, such as an article, then the content data of the multimedia resource can be the text content included in that multimedia resource. If the multimedia resource itself is not a text-type resource, such as a video, image, or product, then the content data of the multimedia resource can be the text content included in the text data describing that multimedia resource. This text data used to describe the multimedia resource can be called the text description data of the multimedia resource. This text description data is used to describe the content of the multimedia resource in text form.

[0055] For example, if a multimedia resource is a product, its text description data can include information describing attributes such as the product's type, color, function, model, and brand. Similarly, if a multimedia resource is an image, its text description data can describe attributes such as the image's type, objects included in the image, its color, size, and source. A multimedia resource can include at least one type of content data. The number of multimedia resources in a multimedia resource set can be determined based on the specific application scenario.

[0056] For example, the content data of a multimedia resource can be a sequence of K sub-content data belonging to that multimedia resource, where K is a positive integer and represents the maximum length of the multimedia resource's content data. If the multimedia resource itself is a text-based resource, then these K sub-content data can be sub-text included within the multimedia resource itself (i.e., the main text). These K sub-content data can be obtained by segmenting the multimedia resource, and the first K segments, i.e., each sub-content data, can be a segment of that multimedia resource. The order of these K segments within the multimedia resource is the same as their order within the multimedia resource's content data.

[0057] If the multimedia resource itself is not a text-type resource, then the K sub-content data can be included in the text description data of the multimedia resource. The K sub-content data can be obtained by segmenting the text description data of the multimedia resource into words. That is, a sub-content data can be a word in the text description data. The order of the K words in the text description data of the multimedia resource is the same as the order in the content data of the multimedia resource.

[0058] For example, if the multimedia resource itself or the text description data of the multimedia resource is segmented, and the number of segments obtained is less than K, then zero elements can be added after the segments to obtain the content data of the multimedia resource, so that the sequence length of the content data is still padded to K.

[0059] The content data of multimedia resources can also be referred to as a word sequence, which can include multiple word sequences. A word sequence can be a segment obtained by segmenting the multimedia resource itself or the text description data of the multimedia resource, or a supplementary 0 element. For example, if K is 6, the content data of a multimedia resource can be the sequence [t1, t2, t3, t4, t5, t6], where t1 to t6 can be 6 segments obtained through word segmentation. As another example, if K is 6, the content data of a multimedia resource can also be the sequence [t1, t2, t3, t4, 0, 0], where t1 to t4 can be 4 segments obtained through word segmentation, and the two 0s after t4 can be supplementary 0 elements to keep the sequence length at 6.

[0060] The processing device can also acquire interaction data of sample objects with respect to multimedia resources in a multimedia resource set. A sample object can be a sample user. There can be at least two sample objects. The number of sample objects can be determined based on the actual application scenario. The processing principle for each sample object is the same. The interaction data of a sample object with respect to multimedia resources can be a sequence consisting of L multimedia resources with interactive behavior of the sample object, where L is a positive integer. L can be the maximum sequence length of the interaction data. One sample object can correspond to at least one piece of interaction data.

[0061] For example, the interactive behavior of the sample object towards multimedia resources could be a click on the multimedia resources, or other behaviors, such as viewing the multimedia resources for a duration exceeding a set time threshold after clicking on them. This application does not impose any restrictions on this. For instance, the L multimedia resources could be the top L multimedia resources with interactive behavior that the sample object had within a recent time period (i.e., a recent time period, such as within a week), such as the top L multimedia resources for which interactive behavior was executed first within that time period. The order of these L multimedia resources in the sample object's interaction data could be the order in which the sample object performed interactive behavior on these L multimedia resources.

[0062] If the number of multimedia resources with which the user has interactive behavior is less than L, the interaction data of the sample object with respect to the multimedia resources can be obtained by padding zero elements after the multimedia resources with which the user has interactive behavior, so that the sequence length of the interaction data is still kept to L.

[0063] For example, if L is 10, the interaction data of a sample object can be the sequence [u1, u2, u3, u4, u5, u6, u7, u8, u9, u10], where u1 to u10 can be 10 multimedia resources for which the user has interactive behavior. The multimedia resources can exist in this sequence through their identifiers. As another example, if L is 10, the content data of a multimedia resource can also be the sequence [u1, u2, u3, u4, u5, u6, 0, 0, 0, 0], where u1 to u6 can be 6 multimedia resources for which the user has interactive behavior, and the 4 zeros after u6 can be supplementary zero elements to keep the sequence length at 10.

[0064] In summary, the content data of multimedia resources can reflect the semantic content of the multimedia resources themselves. Meanwhile, the interaction data of sample objects with multimedia resources can reflect the sample objects' interests and preferences towards those resources. For example, the more frequently a sample object triggers an interaction with a multimedia resource, the more likely it is to belong to a particular content category of multimedia resources that the sample object prefers.

[0065] Step S102: Adaptively train the recommendation model using content data and interaction data to obtain the first recommendation model.

[0066] For example, the processing device can use the content data of multimedia resources in the aforementioned multimedia resource set and the interaction data of sample objects with respect to the multimedia resources in the multimedia resource set to adaptively train the recommendation model. The recommendation model after adaptive training can be called the first recommendation model. This adaptive training can be a comprehensive training of the recommendation model as a whole, enabling the recommendation model to initially learn and perceive the semantic content of multimedia resources and the interest preferences of sample objects with respect to multimedia resources, that is, enabling the recommendation model to initially adapt to the application scenario of recommending multimedia resources to objects in this application. This adaptive training can also be called pre-training in this application.

[0067] For example, the recommendation model to be trained in this application can be a commercially available pre-trained model, such as a publicly available open-source model that already possesses corresponding data processing capabilities, such as the ability to process text data. In this application, the recommendation model can be specifically adjusted in a particular recommendation scenario, such as recommending multimedia resources to an object. That is, training the recommendation model in this application can refer to adjusting the model in a specific direction for a specific application scenario, so that the adjusted recommendation model can be applied to that specific application scenario, such as recommending multimedia resources to an object.

[0068] For example, the recommendation model may include a multimedia model and an object model. The multimedia model can be used to represent multimedia resources, and the object model can be used to represent objects. That is, the multimedia model can be used to generate features of multimedia resources, and the object model can be used to generate features of objects (i.e., users). For example, both the multimedia model and the object model can be LLM models (Large Language Models), meaning that two LLM models can be used to construct the recommendation model in this application. The two LLM models can be identical. Of the two LLM models, one LLM model can be used to represent multimedia resources, and this LLM model can be called the multimedia model; the other LLM model can be used to represent objects, and this LLM model can be called the object model.

[0069] The adaptive training of the recommendation model can include simultaneously correcting the model parameters of both the multimedia model and the object model within the recommendation model. That is, during this training phase, both the model parameters of the multimedia model and the object model need to be trained and corrected. The process of adaptive training of the recommendation model can be described as follows.

[0070] During adaptive training, this application can perform multiple rounds of iterative training on the recommendation model using multimedia resources and sample objects from the multimedia resource set. The multimedia resources and sample objects used in different rounds of iterative training can be different; that is, the content data of the multimedia resources and the interaction data of the sample objects used in different rounds of iterative training can be different. The training principle for each round of iterative training is the same. The following explanation uses the iterative training process of any round of the recommendation model as an example.

[0071] The processing device can invoke a recommendation model (which may be the multimedia model within the recommendation model) to perform feature learning (which may be feature embedding) on ​​the content data of multimedia resources in the multimedia resource set (such as the content data of multimedia resources in any round) to generate multimedia content features of the multimedia resources in the multimedia resource set. These multimedia content features reflect the semantic features of the multimedia resources. Feature learning on the content data of a multimedia resource can generate a corresponding multimedia content feature. The multimedia content features of the multimedia resource can be generated by the multimedia model within the recommendation model.

[0072] For example, the process by which a processing device calls a recommendation model to learn features from the content data of multimedia resources in a multimedia resource set in order to generate multimedia content features of the multimedia resources in the multimedia resource set may include:

[0073] The processing device can invoke the multimedia model within the recommendation model to perform feature learning (such as feature embedding) on ​​each sub-content data in the multimedia resource's content data to generate sub-content features for each sub-content data. A sub-content data can have one sub-content feature. The K sub-content features of the K sub-content data in the content data can constitute a feature sequence (which can be called the fourth feature sequence), and a sub-content feature can be an element in this fourth feature sequence, with that element occupying the position of an element in the fourth feature sequence. The multimedia model in the recommendation model can have an attention mask (which can be called the native attention mask of the multimedia model).

[0074] The multimedia model in the recommendation model can use this native attention mask to apply attention to the elements in the fourth feature sequence to generate multimedia sub-features corresponding to each element position in the fourth feature sequence. The processing device can use the multimedia sub-feature corresponding to the last element position in the fourth feature sequence as the multimedia content feature of the aforementioned multimedia resource. Specifically, the native attention mask can be used to generate the multimedia sub-feature corresponding to the c-th element position based on the element at the c-th element position in the fourth feature sequence and the elements at the element positions preceding the c-th element position. The multimedia sub-feature corresponding to the last element position is a feature generated by fusing the features of the elements at all the preceding element positions, and can serve as an accurate multimedia content feature of the multimedia resource. c is a positive integer, and c is less than or equal to K. In other words, when generating the multimedia sub-features corresponding to each element position in the fourth feature sequence, only the elements at the current element position and the elements at the preceding element positions need to be considered, without needing to consider the elements at the element positions following the current element position.

[0075] The processing device can also invoke a recommendation model to perform feature learning (or feature embedding) on ​​the interaction data of a sample object (such as the interaction data of a sample object in any round with multimedia resources in a multimedia resource set) to generate a first interaction feature for that sample object. This first interaction feature can reflect the sample object's interest in multimedia resources. Performing feature learning on the interaction data of a sample object can generate a corresponding first interaction feature. A first interaction feature can be a feature vector.

[0076] For example, the process by which a processing device calls a recommendation model to perform feature learning on the interaction data of a sample object to generate the first interaction feature of that sample object may include: the processing device may first call the recommendation model (which may be the multimedia model within the recommendation model) to generate multimedia content features of L multimedia resources in the interaction data of the sample object. The multimedia content feature of one of these L multimedia resources may also be obtained by the multimedia model within the recommendation model performing feature learning on the content data of that multimedia resource. A multimedia content feature may be a feature vector.

[0077] The processing device can also invoke the recommendation model to generate the first interaction feature of the sample object using the L multimedia content features of the L multimedia resources. This includes: the L multimedia content features can form a feature sequence of length L (which can be called the third feature sequence). A multimedia content feature can be an element in the third feature sequence, and an element can be located at the position of an element in the third feature sequence. Furthermore, the object model in the recommendation model can have an attention mask (which can be called the native attention mask of the object model).

[0078] The object model in the recommendation model can use this native attention mask to apply attention to the elements in the third feature sequence to generate interactive sub-features corresponding to each element position in the third feature sequence. The processing device can use the interactive sub-feature corresponding to the last element position in the third feature sequence as the first interactive feature of the sample object. Specifically, the native attention mask can be used to generate the interactive sub-feature corresponding to the s-th element position based on the element at the s-th element position in the third feature sequence and the elements at the element positions preceding the s-th element position. The interactive sub-feature corresponding to the last element position is a feature generated by fusing the features of the elements at all preceding element positions, and can be used as the accurate first interactive feature of the sample object. s is a positive integer, and s is less than or equal to L. In other words, when generating the interactive sub-features corresponding to each element position in the third feature sequence, only the elements at the current element position and the elements at the preceding element positions need to be considered, without needing to consider the elements at the element positions following the current element position.

[0079] Therefore, the processing device can obtain the feature similarity (which can be called the first feature similarity) between the multimedia content features of multimedia resources in the multimedia resource set (such as multimedia resources in any round) and the first interaction features of sample objects (which can be sample objects in any round). A multimedia content feature and a first interaction feature can have a feature similarity, that is, a multimedia resource and a sample object can have a feature similarity. Both the multimedia content feature and the first interaction feature can be feature vectors; therefore, the feature similarity between the multimedia content feature and the first interaction feature can be a vector similarity, which can be a cosine similarity.

[0080] The processing device can correct the model parameters of the recommendation model using the first feature similarity to obtain a first recommendation model. For example, the processing device can acquire interaction behavior information of sample objects (such as sample objects from any of the aforementioned rounds) with multimedia resources in the multimedia resource set (such as multimedia resources from any of the aforementioned rounds). A sample object can have interaction behavior information with respect to a multimedia resource, which can be used to indicate whether the sample object has interactive behavior or not with respect to that multimedia resource.

[0081] Therefore, the processing device can obtain the reference feature similarity between a sample object (such as a sample object in any of the aforementioned rounds) and multimedia resources in the multimedia resource set (such as sample objects in any of the aforementioned rounds) based on the interaction behavior information. A sample object and a multimedia resource can have one reference feature similarity. For example, if the interaction behavior information of a sample object towards a multimedia resource indicates that the sample object has interactive behavior towards that multimedia resource, then the reference feature similarity between the sample object and the multimedia resource can be the maximum value of the feature similarity (e.g., it can be 1). Conversely, if the interaction behavior information of a sample object towards a multimedia resource indicates that the sample object does not have interactive behavior towards that multimedia resource, then the reference feature similarity between the sample object and the multimedia resource can be the minimum value of the feature similarity (e.g., it can be 0).

[0082] The processing device can correct the model parameters of the recommendation model by using the difference between the first feature similarity and the reference feature similarity to obtain a first recommendation model. The goal of correcting the model parameters is to minimize (e.g., minimize) the difference between the first feature similarity and the reference feature similarity. For example, this can be achieved using the following formula:

[0083] Where 'a' can represent any multimedia resource (such as any multimedia resource in any round mentioned above), 'b' can represent any sample object (such as any sample object in any round mentioned above), 'A' represents all multimedia resources in any round, and 'B' represents all sample objects in any round. y represents the feature similarity between the first interaction feature of sample object b and the multimedia content feature of multimedia resource a (belonging to the predicted score of the recommendation model). a,b This represents the reference feature similarity between sample object b and multimedia resource a, used to reflect whether sample object b actually has interactive behavior with multimedia resource a.

[0084] The above L BPRThe loss function is generated using the similarity between the first feature and the reference feature. This loss function can be BPR-loss (an interest-based recommendation loss). This loss function represents the feature generation bias of the recommendation model regarding multimedia content features and the first interaction feature. It reflects the difference between the first feature similarity and the reference feature similarity. A larger loss function indicates a greater difference between the first feature similarity and the reference feature similarity. Conversely, a smaller loss function indicates a smaller difference. Therefore, this application can use the loss function to correct the model parameters of the recommendation model to obtain the first recommendation model. Correcting the model parameters can be done by minimizing the value of the loss function (e.g., approaching 0).

[0085] The processing device can use the above principle to perform several rounds of iterative training on the recommendation model (i.e., iteratively correct the model parameters) to finally obtain the first recommendation model. For example, the first recommendation model can be obtained after iteratively training the recommendation model for a specified number of rounds.

[0086] Among them, the multimedia model in the first recommendation model is the multimedia model after adaptive training of the multimedia model in the recommendation model, and the object model in the first recommendation model is the object model after adaptive training of the object model in the recommendation model.

[0087] Step S103: Obtain collaborative features. Collaborative features include at least one of the features extracted from the interaction data that indicate the correlation between objects, between objects and resources, and between resources. Collaborative features include at least one of the following: multimedia collaborative features of multimedia resources in the multimedia resource set, and object collaborative features of sample objects. The multimedia collaborative features are used to represent the content representation of multimedia resources in the multimedia resource set in an online recommendation scenario, and the object collaborative features are used to represent the object representation of sample objects in an online recommendation scenario.

[0088] For example, processing devices can acquire collaborative features. Collaborative features refer to implicit features accumulated in online recommendation scenarios that characterize "multimedia resource relevance" or "user interest similarity." Implicit features or implicit information refer to features or information that can only be obtained after analyzing, processing, and mining data. For example, collaborative features can be implicit attribute vectors strongly associated with multimedia resources or users, generated through learning from historical interaction data in online recommendation scenarios. Collaborative features are generated entirely based on user interaction records with multimedia resources in online recommendation scenarios (such as clicks, browsing, and favorites), such as behavioral data like "User A clicked on technology articles multiple times" or "Article B is frequently viewed by users who like digital products," without relying on textual information such as article titles or user profiles.

[0089] Collaborative features are used to supplement recommendation models with non-textual interaction information, complementing the textual semantic information learned by Large Language Models (LLMs) and ultimately improving recommendation accuracy. Large Language Models (LLMs) can understand resource content through text (e.g., "This article reviews a mobile phone"), while collaborative features can uncover implicit associations (e.g., "This mobile phone review article is frequently clicked by users who have purchased the phone"). Combining the two makes recommendations more accurate.

[0090] The collaborative feature can include at least one of the following: multimedia collaborative features of multimedia resources in a multimedia resource set, and object collaborative features of sample objects. In other words, the collaborative feature can include only multimedia collaborative features of multimedia resources in a multimedia resource set, or only object collaborative features of sample objects, or both multimedia collaborative features and object collaborative features. Multimedia collaborative features are used to represent the "content association attributes" of multimedia resources in online recommendation scenarios, i.e., the implicit association between resources and resources, and between resources and users. For example, the collaborative feature of an article (multimedia resource) can represent implicit information such as "frequently clicked by men aged 30-40" and "highly associated with mobile phone review articles." Object collaborative features are used to represent the "interest preference attributes" of users in online recommendation scenarios, i.e., the implicit association between users and resources, and between users. For example, the collaborative feature of a user (object) can represent implicit information such as "clicked AI-related content 10 times in the past month" and "80% similarity in interests with users who like technology."

[0091] This multimedia collaborative feature can be used to represent the content representation of multimedia resources in an online recommendation scenario within a multimedia resource set. In other words, this multimedia collaborative feature can be an existing feature of the multimedia resources in the online recommendation scenario. This multimedia collaborative feature can be a feature (such as an embedding) learned and generated from the multimedia resources within this online recommendation scenario. For example, it can be a feature learned and generated from the multimedia resources using an existing recall model (such as a fine-grained GNN model) within this online recommendation scenario. The training principle of this recall model needs to be determined based on the specific data recommendation strategy within its online recommendation scenario.

[0092] In this online recommendation scenario, multimedia resources can be represented by their IDs (identifiers). Therefore, the multimedia collaborative feature can be the ID collaborative feature of the multimedia resource. This ID collaborative feature can be obtained by feature embedding processing of the multimedia resource ID through the recall model. The ID collaborative feature can include rich collaborative information of the multimedia resource in the online recommendation scenario. This collaborative information can include any information displayed by the multimedia resource in the online recommendation scenario, such as information interacted with by the user, attribute information of the multimedia resource in the online recommendation scenario, other information related to the multimedia resource in the online recommendation scenario, etc. The online recommendation scenario in which the multimedia resource is located can be called the first online recommendation scenario. For example, the multimedia resource can be an article, and the first online recommendation scenario can be the article recommendation scenario in the subscription account where the article is located; or, for example, the multimedia resource can be a short video, and the first online recommendation scenario can be the video recommendation scenario in the video client where the short video is located; and so on.

[0093] The aforementioned object collaborative features can be used to represent the object representation of a sample object in an online recommendation scenario. That is, these object collaborative features can be features already present in the online recommendation scenario for the sample object. These object collaborative features can be features (such as embeddings) learned and generated from the sample object in this online recommendation scenario. For example, they can be features learned and generated from the sample object using an existing recall model in this online recommendation scenario (such as a fine-grained ranking GNN model). The training principle of this recall model needs to be determined based on the data recommendation strategy of the online recommendation scenario in which it is located.

[0094] In this online recommendation scenario, sample objects can be represented by their IDs (identifiers). Therefore, the collaborative feature of the object can be the ID collaborative feature of the sample object. This ID collaborative feature can be obtained by feature embedding processing of the sample object's ID through the recall model. The ID collaborative feature can include rich collaborative information of the sample object in the online recommendation scenario. This collaborative information can include any information exhibited by the sample object in the online recommendation scenario, such as preference information for interacting with the data recommended in the online recommendation scenario, attribute information of the sample object in the online recommendation scenario, other information related to the sample object in the online recommendation scenario, etc. The online recommendation scenario in which the sample object is located can be called the second online recommendation scenario. There can be at least one second online recommendation scenario, such as P scenarios, where P is a positive integer. For example, the second online recommendation scenario can be a scenario of recommending friends' updates to users in WeChat Moments, or a scenario of recommending products to users in a shopping platform, or a scenario of recommending short videos to users in a short video platform. The second online recommendation scenario and the first online recommendation scenario can be different online recommendation scenarios. An online recommendation scenario can refer to a scenario that has already been put into application and is already used for online data recommendation to objects.

[0095] Step S104: The first recommendation model is optimized and trained using collaborative features to obtain the trained recommendation model; the trained recommendation model is used to recall (i.e. determine or select) multimedia resources recommended to the object.

[0096] For example, the processing device can use the acquired collaborative features to optimize and train the first recommendation model to obtain the final trained recommendation model. Specifically, if the collaborative features only include multimedia collaborative features, it indicates that the first recommendation model can be optimized and trained by introducing only multimedia collaborative features without introducing object collaborative features. If the collaborative features only include object collaborative features, it indicates that the first recommendation model can be optimized and trained by introducing only object collaborative features without introducing multimedia collaborative features. If the collaborative features include both multimedia and object collaborative features, it indicates that the first recommendation model can be optimized and trained by introducing both multimedia and object collaborative features simultaneously. All three methods of optimizing and training the first recommendation model fall within the scope of protection of this application.

[0097] The specific features included in the collaborative features can be determined based on the actual application scenario. For example, in scenarios where saving training costs is crucial (or where processing equipment performance is low), if a deeper understanding of multimedia resource features is required, then the collaborative features may only include multimedia collaborative features. Similarly, in scenarios where saving training costs is crucial, if a deeper understanding of sample object features is required, then the collaborative features may only include object collaborative features. However, if saving training costs is not a concern, and a deeper understanding of both multimedia resource and sample object features is needed, then the collaborative features may include both multimedia collaborative features and object collaborative features.

[0098] The following section will use the example of collaborative features that include both multimedia collaborative features and object collaborative features to describe how to optimize and train the first recommendation model.

[0099] The processing device can first use the aforementioned multimedia collaborative features to optimize and train the first recommendation model (also known as fine-tuning, here referring to the first fine-tuning). This optimized first recommendation model can be called the second recommendation model. This second recommendation model is obtained by optimizing and training the multimedia collaborative features of multimedia resources. The optimization and training of the first recommendation model can include: simultaneously correcting the model parameters of the multimedia model and the object model within the first recommendation model; that is, during the optimization and training process of the first recommendation model, the model parameters of both the multimedia model and the object model need to be corrected.

[0100] The multimedia model in the second recommendation model is the multimedia model obtained by optimizing and training the multimedia model in the first recommendation model. The object model in the second recommendation model is the object model obtained by optimizing and training the object model in the first recommendation model. The process of optimizing and training the first recommendation model through multimedia collaborative features to obtain the second recommendation model can be seen in the description of the embodiment corresponding to Figure 5 below.

[0101] The processing device can use the aforementioned object collaborative features to further optimize and train the second recommendation model (also known as fine-tuning, here referring to a second fine-tuning). This optimized second recommendation model can be used as the final trained recommendation model. This final trained recommendation model is the adjusted recommendation model. This trained recommendation model is obtained by incorporating the object collaborative features of the sample objects for optimization training.

[0102] Optimizing the training of the second recommendation model can include: keeping the model parameters of the multimedia model in the second recommendation model unchanged (e.g., freezing the model parameters of the multimedia model in the second recommendation model without modification), and modifying the model parameters of the object model in the second recommendation model. That is, during the optimization training of the second recommendation model, the model parameters of the multimedia model do not need to be modified, only the model parameters of the object model need to be modified.

[0103] Because the object model generates object features based on features generated from multimedia resources by the multimedia model, during the optimization training of the object model by introducing collaborative object features, the multimedia model is also undergoing optimization training. This causes changes in the features generated by the multimedia model from multimedia resources, making the object model's generation of object features based on these features unstable and ultimately resulting in poor optimization training performance. Therefore, this application, when optimizing the second recommendation model, eliminates the need to modify the multimedia model's parameters; instead, it only modifies the object model's parameters, thus ensuring the stability and superiority of the object model's optimization training performance.

[0104] In cases where collaborative features include both multimedia collaborative features and object collaborative features, this application employs an optimization sequence of first introducing multimedia collaborative features to optimize and train the first recommendation model, and then introducing object collaborative features to optimize and train the second recommendation model. This sequence is based on specific reasons and considerations. By first introducing multimedia collaborative features to optimize and train the first recommendation model, and then fixing the model parameters of the multimedia model within the first recommendation model after optimization and training, this application achieves the fixation of the features generated from multimedia resources through the multimedia model. Consequently, when introducing object collaborative features to optimize and train the second recommendation model, the fixed features of the multimedia resources can be used to stably generate object features in the object model, thereby achieving stable and accurate training of the object model.

[0105] The multimedia model in the trained recommendation model is the multimedia model optimized and trained from the multimedia model in the second recommendation model, and the object model in the trained recommendation model is the object model optimized and trained from the object model in the second recommendation model. The process of optimizing and training the second recommendation model using object collaborative features to obtain the trained recommendation model can be seen in the description of the embodiment corresponding to Figure 8 below.

[0106] For example, if the collaborative features only include multimedia collaborative features, this application can directly use the second recommendation model, which is optimized and trained using the multimedia collaborative features, as the final trained recommendation model. If the collaborative features only include object collaborative features, this application can directly introduce object collaborative features to optimize and train the first recommendation model, and the optimized and trained first recommendation model in this case can be directly used as the final trained recommendation model. Furthermore, regardless of which features the collaborative features include or which training stage the recommendation model is in, the principle for training the model can be the principle shown in formula (1) above.

[0107] The introduction of multimedia collaborative features in this application affects the way the multimedia model generates features for multimedia resources, while the introduction of object collaborative features affects the way the object model generates features for objects. Therefore, if the collaborative features include multimedia collaborative features, then when only multimedia collaborative features are introduced for optimization training of the first recommendation model, the principle by which the multimedia model generates features for multimedia resources can be the same as the principle by which features for multimedia resources (such as the multimedia fusion features described below) are generated in the embodiment corresponding to Figure 5 below. Since object collaborative features are not introduced, the principle by which the object model generates features for objects (such as the second interaction features described below) can be the same as the principle by which the object model generates features for objects during adaptive training.

[0108] Similarly, if the collaborative features include object collaborative features, when only object collaborative features are introduced to optimize and train the first recommendation model, the first recommendation model can be used as the second recommendation model, and the object collaborative features can be used to optimize and train the second recommendation model to obtain the trained recommendation model. The principle of the object model generating object features can be the same as the principle of generating object features (such as the object fusion features described below) in the embodiment corresponding to Figure 8 below. Since multimedia collaborative features are not introduced, the principle of the multimedia model generating multimedia resource features can be the same as the principle of the multimedia model generating multimedia resource features (such as the multimedia content features mentioned above) during adaptive training.

[0109] After training the aforementioned recommendation model, this application can apply the trained recommendation model to recall multimedia resources recommended to an object. That is, the trained recommendation model in this application can be used to recommend multimedia resources that match the object's interests. The specific process of this recommendation can be found in the relevant description in the embodiment corresponding to Figure 10 below. The processing device can push the multimedia resources recalled by the trained recommendation model to the user's client for viewing multimedia resources, so that the client can display the recommended multimedia resources to the user in the recommendation interface (which belongs to the client interface) for the user to view.

[0110] Figure 4 is a schematic diagram illustrating the effect of a multimedia resource recommendation interface according to an embodiment of this application. As shown in Figure 4, the multimedia resource here can be an article. The recommendation interface can display a subscription recommendation list, which sequentially displays various multimedia resources (i.e., articles) recalled for the user, including the article "Special Topic Interpretation | Latest Research Progress in Large Language Models", the article "Simultaneous Training of Multiple Graph Neural Networks", and the article "The Market Value of xxx Company Soars", thereby realizing the recommendation of multimedia resources for the user.

[0111] Furthermore, the first recommendation model, the second recommendation model, and the trained recommendation model mentioned above are actually the same model at different stages. Also, since the model parameters of the object model are only adjusted during the optimization training of the second recommendation model, not the multimedia model's parameters, the multimedia model in the second recommendation model can be the same as the multimedia model in the trained recommendation model.

[0112] By employing the method described in this application, progressive training of the recommendation model is achieved through three stages: adaptive training, optimization training by introducing multimedia collaborative features, and optimization training by introducing object collaborative features. This allows for the gradual and in-depth introduction of basic features, multimedia collaborative features, and object collaborative features into the recommendation model, enabling the model to learn multimedia collaborative features and object collaborative features more adaptably and deeply. Ultimately, a high-performing recommendation model is trained, which can accurately recall and recommend multimedia resources to users.

[0113] This application can acquire content data of multimedia resources in a multimedia resource set and interaction data of sample objects with respect to multimedia resources in the multimedia resource set; it can also use the content data and interaction data to adaptively train a recommendation model to obtain a first recommendation model; and it can acquire collaborative features; the collaborative features include at least one of the following: multimedia collaborative features of multimedia resources in the multimedia resource set and object collaborative features of sample objects, wherein the multimedia collaborative features are used to represent the content representation of multimedia resources in the multimedia resource set in an online recommendation scenario, and the object collaborative features are used to represent the object representation of sample objects in an online recommendation scenario; and the first recommendation model can be optimized and trained using the collaborative features to obtain a trained recommendation model; the trained recommendation model is used to recall multimedia resources recommended to objects. As can be seen, the method proposed in this application can perform multi-stage progressive training on the recommendation model. In this multi-stage progressive training, the recommendation model can first undergo overall adaptive training, allowing it to initially learn information related to multimedia resources and objects. Furthermore, collaborative features (such as multimedia collaborative features of multimedia resources and / or object collaborative features of sample objects) can be introduced to further optimize the training of the recommendation model, enabling it to learn the existing features of multimedia resources and / or sample objects in online recommendation scenarios. This can greatly improve the training effect of the recommendation model, resulting in a superior recommendation model. The trained recommendation model can then accurately recall multimedia resources recommended to objects.

[0114] Figure 5 is a schematic flowchart illustrating the optimization training of a first recommendation model according to an embodiment of this application. As shown in Figure 5, the process may include the following steps.

[0115] Step S201: Call the multimedia model in the first recommendation model to learn features for each sub-content data in the content data and generate sub-content features for each sub-content data.

[0116] For example, since the content data of multimedia resources in a multimedia resource set can be a sequence composed of K sub-content data of the respective multimedia resource, the processing device can call the multimedia model in the first recommendation model to perform feature learning (such as feature embedding) on ​​each sub-content data in the content data (such as the content data of multimedia resources in any of the aforementioned rounds) to generate sub-content features for each sub-content data. A sub-content data can have one sub-content feature, and a sub-content feature can be a feature vector. By performing feature learning on K sub-content data in a content data set, K sub-content features of those K sub-content data can be generated, and these K sub-content features can be used to characterize the content semantics of the respective multimedia resource.

[0117] Step S202: The multimedia collaborative features and the K sub-content features of the K sub-content data are sequentially concatenated to obtain the first feature sequence.

[0118] For example, the processing device can sequentially concatenate the multimedia collaborative features and the K sub-content features of the K sub-content data to obtain a first feature sequence. The multimedia collaborative features and the K sub-content features concatenated in this first feature sequence all belong to the same multimedia resource. Therefore, one multimedia resource can correspond to one first feature sequence, and the principles for processing each multimedia resource are the same and independent.

[0119] When there are m multimedia collaborative features, the multimedia collaborative features and K sub-content features are the K+m elements in the first feature sequence, where m is a positive integer. In the first feature sequence, the K sub-content features are located after the m multimedia collaborative features. For example, in some cases, a multimedia resource has only one multimedia collaborative feature, which is the feature of that multimedia resource in the online recommendation scenario. Therefore, the multimedia collaborative features and K sub-content features concatenated in the first feature sequence are the K+1 elements in the first feature sequence, and one of these K+1 elements is located at an element position in the first feature sequence. The first feature sequence includes a total of K+1 element positions, and the multimedia collaborative features can be located at the first element position in the first feature sequence (i.e., the first element position). That is, the sub-content features are located after the multimedia collaborative features in the first feature sequence. The concatenation order of the K sub-content features in the first feature sequence is the order of the K sub-content data to which the K sub-content features belong in the corresponding content data.

[0120] For example, the K sub-content features can sequentially include sub-content feature 1 to sub-content feature 6. Then, a first feature sequence can be [multimedia collaboration feature, sub-content feature 1, sub-content feature 2, sub-content feature 3, sub-content feature 4, sub-content feature 5, sub-content feature 6]. The multimedia collaboration features in this first feature sequence, as well as the content data to which sub-content features 1 to sub-content features 6 belong, can all be multimedia collaboration features and content data of the same multimedia resource.

[0121] Step S203: Optimize and train the first recommendation model based on the first feature sequence to obtain the second recommendation model.

[0122] For example, the processing device can invoke the multimedia model in the first recommendation model to generate multimedia fusion features of multimedia resources in the multimedia resource set through the first feature sequence. A multimedia resource can have a multimedia fusion feature, which is a feature obtained by fusing the multimedia collaborative features of the multimedia resource and the content features (such as K sub-content features) generated by the multimedia model for that multimedia resource.

[0123] The process of generating the multimedia fusion feature may include: the multimedia model in the first recommendation model may have a first attention mask, and the processing device may call the multimedia model in the first recommendation model to perform attention processing on the elements in the first feature sequence through the first attention mask to generate the fusion sub-feature corresponding to each element position in the first feature sequence. A fusion sub-feature may also be a feature vector.

[0124] Therefore, the processing device can use the fusion sub-feature corresponding to the last element position (i.e., the last element position) in the first feature sequence as the multimedia fusion feature of the multimedia resources in the multimedia resource set. Because the fusion sub-feature corresponding to the last element position is a feature obtained by fusing the sub-content features at each of the previous element positions, the fusion sub-feature corresponding to the last element position can be used to characterize the features of the first feature sequence as a whole.

[0125] The aforementioned first attention mask is an innovative and specifically added attention mask in the multimedia model during the optimization training phase of the first recommendation model in this application. This attention mask can be used to generate fused sub-features corresponding to the positions of each element in the first feature sequence through the multimedia collaborative features concatenated in the first feature sequence.

[0126] For example, using this first attention mask, the fusion sub-feature corresponding to the i-th element position can be generated by using the element at the i-th element position in the first feature sequence and the elements at the element positions preceding the i-th element position. i is a positive integer, and i is less than or equal to K+1. The i-th element position can be any element position in the first feature sequence, meaning that when generating the fusion sub-feature corresponding to an element position in the first feature sequence, this application can focus on the elements at that element position and the elements preceding it, without considering the elements at the element positions following that element position. Since the multimedia collaborative feature is located at the first element position in the first feature sequence, the fusion sub-features corresponding to each element position of each sub-content feature in the first feature sequence are all generated in conjunction with this multimedia collaborative feature, thus achieving the effect of introducing the multimedia collaborative feature into the fusion sub-features corresponding to each element position.

[0127] Figure 6 is a schematic diagram of the structure of a first attention mask according to an embodiment of this application. As shown in Figure 6, the dimension of the first attention mask is (K+1)*(K+1), that is, the square of K+1, where K can be equal to 4. The token sequence here can refer to the aforementioned content data, where a sub-content data has a sub-content feature. The element position of a sub-content feature in the first feature sequence can correspond to a row and a column in the first attention mask. For example, any element in the first attention mask can be used to obtain the correlation between any two element positions in the first feature sequence. Each row and each column in the first attention mask corresponds sequentially to each element position in the first feature sequence. For example, the first row and the first column in the first attention mask can both correspond to the first element position in the first feature sequence, the second row and the second column in the first attention mask can both correspond to the second element position in the first feature sequence, and so on.

[0128] In this first attention mask, an element that is 1 indicates that the correlation between the two corresponding element positions in the first feature sequence needs to be acquired, while an element that is 0 indicates that the correlation between the two corresponding element positions in the first feature sequence does not need to be acquired (or does not need to be considered). This can be understood as the correlation between the two corresponding element positions in the first feature sequence being infinite or zero. Here, an element that is 0 in the first attention mask means that the correlation between the current element position and the element positions following it in the first feature sequence does not need to be acquired.

[0129] For example, the first row and first column of the first attention mask correspond to the position of the first element in the first feature sequence (i.e., the position of the element where the multimedia collaborative feature is located). The element 1 in the first row and first column of the first attention mask can be used to obtain the correlation between the multimedia collaborative feature and itself.

[0130] For example, the second row and first column of the first attention mask correspond to the second element position and the first element position in the first feature sequence. The element 1 in the second row and first column of the first attention mask is used to obtain the correlation between the second element position (i.e. the element position where the first sub-content feature is located) and the first element position (i.e. the element position where the multimedia collaboration feature is located) in the first feature sequence.

[0131] For example, the third row and second column of the first attention mask corresponds to the third element position and the second element position in the first feature sequence. The element 1 in the third row and second column of the first attention mask is used to obtain the correlation between the third element position (i.e., the element position where the second sub-content feature is located in the first feature sequence) and the second element position (i.e., the element position where the first sub-content feature is located in the first feature sequence).

[0132] As shown in Figure 6, element 1 in the first row and element 1 in the first column of the first attention mask can be used to obtain the relevance related to multimedia collaborative features, while element 1 in the gray part of Figure 6 can be used to obtain the relevance between the sub-content features of each sub-content data in the Token sequence.

[0133] For example, the fusion sub-feature corresponding to the i-th element position in the first feature sequence can be generated by weighted summation of the elements at the i-th element position and the elements at the i-th element position and the elements at the i-th element position before the i-th element position, based on the correlation between the i-th element position and the i-th element position itself and the elements at the i-th element position before the i-th element position. The correlation can be used as the weight.

[0134] The processing device can also invoke the object model in the first recommendation model to perform feature learning on the interaction data of the sample object to generate a second interaction feature of the sample object. This process may include: the processing device can first invoke the multimedia model in the first recommendation model to generate multimedia fusion features of L multimedia resources in the interaction data. The principle of generating the L multimedia fusion features of the L multimedia resources in the interaction data is the same as the principle of invoking the multimedia model in the first recommendation model to generate multimedia fusion features of multimedia resources in the multimedia resource set. A multimedia fusion feature can be a feature vector. Thus, the processing device can invoke the object model in the first recommendation model to generate the second interaction feature of the sample object through the native attention mask of the object model and the L multimedia fusion features. The principle of invoking the object model in the first recommendation model to generate the second interaction feature of the sample object through the native attention mask of the object model and the L multimedia fusion features is the same as the principle of invoking the object model in the recommendation model to generate the first interaction feature of the sample object through the native attention mask of the object model and the L multimedia content features, and will not be elaborated here. In this context, a sample object can have a second interaction feature, which can be a feature vector.

[0135] The processing device can obtain the feature similarity between the multimedia fusion features of multimedia resources (which may include all multimedia resources in any round) in a multimedia resource set and the second interaction features of sample objects (which may include all sample objects in any round). This feature similarity can be called the second feature similarity. A multimedia resource and a sample object can have one second feature similarity. The calculation principle of this second feature similarity can be the same as that of the first feature similarity, such as the cosine similarity between feature vectors. The processing device can use this second feature similarity to correct the model parameters of the first recommendation model to obtain the second recommendation model.

[0136] The principle of correcting the model parameters of the first recommendation model through the second feature similarity to obtain the second recommendation model is the same as the principle of correcting the model parameters of the recommendation model through the first feature similarity to obtain the first recommendation model. For example, both can be based on the principle represented by the above formula (1). Furthermore, the first recommendation model can be trained through several rounds of iterative training. The multimedia resources used in each round of iterative training (i.e., the corresponding content data and multimedia collaborative features) can be different, or they can be partially the same. The sample objects used in each round of iterative training (i.e., the corresponding interaction data) can also be different, or they can be partially the same, depending on the actual application scenario.

[0137] Figure 7 is a schematic diagram of a scenario for optimizing and training a first recommendation model according to an embodiment of this application. As shown in Figure 7, the Item Modeling (LLM) in the first recommendation model can be a multimedia model, and the User Modeling (LLM) in the first recommendation model can be an object model. When optimizing and training the first recommendation model, this application can incorporate multimedia collaborative features into the Item Modeling (LLM) for training. For example, the Item Modeling (LLM) can be called to generate multimedia fusion features of multimedia resources in one round using multimedia collaborative features. The multimedia fusion features generated by the Item Modeling (LLM) from the multimedia resources in the interaction data of the sample object using multimedia collaborative features can be input into the User Modeling (LLM). Therefore, in the User Modeling (LLM), the second interaction features of the sample object can be generated using the multimedia fusion features of the multimedia resources in the interaction data of the sample object.

[0138] Therefore, the processing device can correct the model parameters of the first recommendation model by the difference between the multimedia fusion features of the multimedia resources in this round and the second interaction features of the sample objects (which can be reflected by the similarity of the second features mentioned above).

[0139] This application proposes an effective masking strategy (such as a feature fusion strategy based on the aforementioned first attention mask), which can effectively incorporate multimedia collaborative features of multimedia resources for training multimedia models. Since multimedia collaborative features include rich collaborative information from multimedia resources (such as various information exhibited by multimedia resources in online recommendation scenarios, and various operational information triggered by users in online recommendation scenarios), multimedia models trained in this way can possess extremely strong representational capabilities for multimedia resources.

[0140] Figure 8 is a schematic flowchart of an embodiment of this application for optimizing and training a second recommendation model. As shown in Figure 8, the process may include the following steps.

[0141] Step S301: Obtain the multimedia fusion features of each of the L multimedia resources. The L multimedia fusion features of the L multimedia resources are generated by calling the multimedia model in the second recommendation model.

[0142] For example, since the interaction data of a sample object is a sequence composed of L multimedia resources with interactive behaviors, the processing device can obtain the multimedia fusion features of each of the L multimedia resources. These L multimedia fusion features are generated by calling the multimedia model in the second recommendation model. The principle behind generating these L multimedia fusion features is the same as the principle behind generating the multimedia fusion features of the multimedia resources in the multimedia resource set in step S203 above.

[0143] Step S302: The object collaborative features are concatenated with the L multimedia fusion features in the form of a prefix of the L multimedia fusion features to obtain the second feature sequence.

[0144] For example, this application employs a Prefix-tuning efficient fine-tuning paradigm to fuse the object collaboration features of sample objects. The object collaboration features of the sample object can be concatenated with the aforementioned L multimedia fusion features in a prefix form (Prefix form) to obtain a second concatenated sequence. One sample object can correspond to one second concatenated sequence, and the object collaboration features concatenated in the second concatenated sequence, as well as the interaction data to which the L multimedia fusion features concatenated in the second concatenated sequence belong, can all belong to the same sample object.

[0145] In this context, a sample object can exist in at least one online recommendation scenario, and in each online recommendation scenario, it can possess one object collaborative feature. Therefore, a sample object can also have at least one object collaborative feature, such as P object collaborative features, where P is a positive integer (i.e., an integer greater than or equal to 1). For example, multimedia resources can be articles, and this application can be applied to article recommendation scenarios. The P online recommendation scenarios in which the sample object exists can be other recommendation scenarios besides the article recommendation scenario, such as video recommendation scenarios and friend activity recommendation scenarios, etc.

[0146] Therefore, the P object collaborative features and L multimedia fusion features can be P+L elements in the second feature sequence, meaning the sequence length of the second feature sequence can be P+L. One of these P+L elements can be located at any position in the second feature sequence, which can have a total of P+L element positions. The P object collaborative features can be located in the first P element positions of the second feature sequence. The order of the P object collaborative features in the second feature sequence can be arbitrary. The order of the L multimedia fusion features in the second feature sequence can be the order of the L multimedia resources to which these L multimedia fusion features belong in the corresponding interactive data.

[0147] For example, P object collaboration features may include object collaboration feature 1 and object collaboration feature 2, and L multimedia fusion features may include multimedia fusion feature 1 to multimedia fusion feature 6. Then, a second feature sequence obtained by concatenating the P object collaboration features and the L multimedia fusion features could be [object collaboration feature 1, object collaboration feature 2, multimedia fusion feature 1, multimedia fusion feature 2, multimedia fusion feature 3, multimedia fusion feature 4, multimedia fusion feature 5, multimedia fusion feature 6]. The interactive data containing the multimedia resources to which object collaboration features 1 and object collaboration features 2, as well as multimedia fusion features 1 to multimedia fusion features 6, all belong to the same sample object.

[0148] Step S303: Add L prefix model parameters corresponding to multimedia fusion features to the object model in the second recommendation model, and correct the prefix model parameters based on the second feature sequence to obtain the trained recommendation model.

[0149] For example, since the object collaboration features of the aforementioned sample object are concatenated with the L multimedia fusion features of the L multimedia resources in the interaction data of the sample object in a prefix form, the processing device can add prefix model parameters corresponding to the L multimedia fusion features to the object model in the second recommendation model. The number of prefix model parameters can be the same as the number of multimedia fusion features concatenated in a prefix form, such as both being L. Each concatenated multimedia fusion feature can correspond to one prefix model parameter. This prefix model parameter can be used to fuse the multimedia fusion feature with other features (such as the concatenated object collaboration features), and can also be used to record the information learned from the multimedia fusion feature during the training process of the object model. This information is reflected through the trained prefix model parameters.

[0150] Therefore, in the process of optimizing and training the second recommendation model in this application, the original model parameters of the second recommendation model can be frozen, and only the newly added prefix model parameters in the second recommendation model can be corrected. Since the proportion of these prefix model parameters in the model parameters of the second recommendation model is less than 0.5%, such as about 0.35%, extremely efficient optimization and training of the second recommendation model can be achieved in this optimization training stage.

[0151] For example, the processing device can use the aforementioned second feature sequence to correct the parameters of the newly added prefix model in the second recommendation model, thereby obtaining the trained recommendation model. This process may include:

[0152] The processing device can invoke the multimedia model in the second recommendation model to generate multimedia fusion features of multimedia resources (such as multimedia resources in any round) in the multimedia resource set by using the content data and multimedia collaboration features of the multimedia resources in the multimedia resource set. A multimedia resource can have one multimedia fusion feature. The principle of invoking the multimedia model in the second recommendation model to generate multimedia fusion features of multimedia resources is the same as the principle of invoking the multimedia model in the first recommendation model to generate multimedia fusion features of multimedia resources through the first feature sequence, and will not be elaborated further here.

[0153] The processing device can also invoke a second recommendation model with newly added prefixed model parameters to generate object fusion features for sample objects (which can be sample objects from any round) using the aforementioned second feature sequence. A sample object can have one object fusion feature. This process can be described as follows.

[0154] The object model in the second recommendation model can have a second attention mask. The processing device can invoke the object model in the second recommendation model to perform attention processing on the elements in the second feature sequence through this second attention mask, so as to generate a fused sub-feature corresponding to each element position in the second feature sequence. Here, a fused sub-feature can also be a feature vector.

[0155] Therefore, the processing device can use the fusion sub-feature corresponding to the last element position (i.e., the last element position) in the second feature sequence as the object fusion feature of the sample object. Because the fusion sub-feature corresponding to the last element position is a feature obtained by fusing the elements at the previous element positions, the fusion sub-feature corresponding to the last element position can be used to characterize the fusion feature of the second feature sequence as a whole.

[0156] The aforementioned second attention mask is an innovative and specifically added attention mask in the object model during the optimization training stage of the second recommendation model in this application. This attention mask can be used to generate fused sub-features corresponding to the positions of each element in the second feature sequence by using the object collaborative features concatenated in the second feature sequence.

[0157] For example, using this second attention mask, the fusion sub-feature corresponding to the j-th element position can be generated by using the element at the j-th element position in the second feature sequence and the elements at the element positions preceding the j-th element position. j is a positive integer, and j is less than or equal to P+L. The j-th element position can be any element position in the second feature sequence. This means that when generating the fusion sub-feature corresponding to an element position in the second feature sequence, this application can focus on the elements at that element position and the element positions preceding it, without considering the elements at the element positions following that element position. Since the object collaborative feature is located at the first P element positions in the second feature sequence, the fusion sub-features corresponding to each element position of the L multimedia fusion features in the second feature sequence are all generated in combination with the object collaborative feature, thereby achieving the effect of introducing the object collaborative feature into the fusion sub-features corresponding to each element position.

[0158] Figure 9 is a schematic diagram of the structure of a second attention mask according to an embodiment of this application. As shown in Figure 9, the dimension of the first attention mask can be (P+L)*(P+L), that is, the square of P+L. Here, P can be equal to 2, and L can be equal to 4. The article sequence here can refer to the above-mentioned interactive data, and the multimedia resource can be an article. One multimedia resource in the interactive data has a multimedia fusion feature. The element position of a multimedia fusion feature in the second feature sequence can correspond to a row and a column in the second attention mask. For example, any element in the second attention mask can be used to obtain the correlation between any two element positions in the second feature sequence. Each row and each column in the second attention mask corresponds to each element position in the second feature sequence in turn. For example, the first row and the first column in the second attention mask can both correspond to the first element position in the second feature sequence, the second row and the second column in the first attention mask can both correspond to the second element position in the second feature sequence, and so on.

[0159] In this context, elements in the second attention mask that are 1 indicate that the correlation between the two corresponding element positions in the second feature sequence needs to be acquired. Elements in the first attention mask that are 0 indicate that the correlation between the two corresponding element positions in the second feature sequence does not need to be acquired (or does not need to be considered), which can be understood as the correlation between the two corresponding element positions in the second feature sequence being infinite or 0. Here, an element of 0 in the second attention mask means that the correlation between the current element position and the element positions following it in the second feature sequence does not need to be acquired.

[0160] Since there are two object collaborative features in this example, in the first and second rows (i.e., the first P rows) of the second attention mask, all elements except the first two, which are 1, are 0. This is because the elements in the first and second rows of the second attention mask are used to fuse P object collaborative features, thus requiring the calculation of the correlation between these P object collaborative features. In other words, the elements in the first P rows and first P columns of the second attention mask can be 1, while the elements in the first P rows except for the first P columns, and the elements in the first P columns except for the first P rows, can all be 0. This design is for fusing the P object fusion features.

[0161] For example, the first row and first column of the second attention mask correspond to the position of the first element in the second feature sequence (i.e., the position of the element where the first object collaborative feature is located). The element 1 in the first row and first column of the second attention mask can be used to obtain the correlation between the first object collaborative feature and itself.

[0162] For example, the second row and first column of the second attention mask corresponds to the second element position and the first element position in the second feature sequence. The element 1 in the second row and first column of the second attention mask is used to obtain the correlation between the second element position (i.e., the element position where the second object co-feature is located) and the first element position (i.e., the element position where the first object co-feature is located) in the second feature sequence.

[0163] For example, the fourth row and third column of the second attention mask corresponds to the fourth and third element positions in the second feature sequence. The element 1 in the fourth row and third column of the second attention mask is used to obtain the correlation between the fourth element position (i.e., the element position where the second multimedia fusion feature is located) and the third element position (i.e., the element position where the first multimedia fusion feature is located) in the second feature sequence.

[0164] As shown in Figure 9, the elements 1 in the first and second columns of the second attention mask can be used to obtain the relevance related to the collaborative features of the object, while the elements 1 in the gray part of Figure 9 can be used to obtain the relevance between the multimedia fusion features of each multimedia resource in the article sequence.

[0165] For example, the fusion sub-feature corresponding to the j-th element position in the first feature sequence can be generated by weighted summation of the elements at the j-th element position and the elements at the j-th element position and the elements at the element positions before the j-th element position based on the correlation between the j-th element position and the j-th element position itself and the element positions before the j-th element position. The correlation can be used as the weight.

[0166] The processing device can obtain the third feature similarity between the multimedia fusion features of multimedia resources (which can be multimedia resources from any round) in the multimedia resource set and the object fusion features of sample objects (which can be sample objects from any round). A multimedia resource and a sample object can have one third feature similarity. The calculation principle of this third feature similarity can be the same as that of the first feature similarity, such as the cosine similarity between feature vectors. The processing device can use this third feature similarity to correct the newly added prefix model parameters in the second recommendation model to obtain the trained recommendation model.

[0167] The principle of correcting the prefix model parameters through the third feature similarity to obtain the trained recommendation model is the same as the principle of correcting the recommendation model parameters through the first feature similarity to obtain the first recommendation model. For example, both can be based on the principle expressed by formula (1) above. Furthermore, the second recommendation model can undergo several rounds of iterative training. The multimedia resources used in each round of iterative training (i.e., the corresponding content data and multimedia collaborative features) can be different, or partially the same. The sample objects used in each round of iterative training (i.e., the corresponding interaction data) can also be different, or partially the same, depending on the actual application scenario.

[0168] The multimedia resources in the interaction data of the sample objects are used to generate features of the sample objects (such as object fusion features, first interaction features, and second interaction features). No feature similarity (such as first feature similarity, second feature similarity, and third feature similarity) is calculated between the multimedia resources in the interaction data and the sample objects to correct model parameters. However, in each training epoch of each training phase, the multimedia resources used to calculate feature similarity with the sample objects include all multimedia resources in that training epoch.

[0169] Figure 10 is a schematic diagram of a scenario for optimizing and training a second recommendation model according to an embodiment of this application. As shown in Figure 10, the Item Modeling (LLM) in the second recommendation model can be a multimedia model, and the User Modeling (LLM) in the second recommendation model can be an object model. When optimizing and training the second recommendation model, this application can incorporate multimedia collaborative features into the Item Modeling (LLM) and integrate object collaborative features into the User Modeling (LLM) for training. For example, the Item Modeling (LLM) can be called to generate multimedia fusion features of multimedia resources for one round using multimedia collaborative features. The multimedia fusion features generated by the Item Modeling (LLM) from the multimedia resources in the interaction data of sample objects using multimedia collaborative features can be input into the User Modeling (LLM). Therefore, the User Modeling (LLM) can generate object fusion features of sample objects using the multimedia fusion features of multimedia resources in the interaction data of sample objects and the object collaborative features.

[0170] Therefore, the processing device can correct the model parameters of the second recommendation model by the difference between the multimedia fusion features of the multimedia resources in this round and the object fusion features of the sample objects (which can be reflected by the third feature similarity mentioned above), such as correcting the prefix model parameters added in the form of Prefix in the second recommendation model.

[0171] The recommendation model trained as described above is the final recommendation model obtained through progressive training. This application proposes an effective masking strategy (such as a feature fusion strategy based on the aforementioned second attention mask). This masking strategy can effectively incorporate the collaborative features of sample objects to train the object model. Since the collaborative features of objects include rich collaborative information about the sample objects (such as various information exhibited by the sample objects in online recommendation scenarios, such as the sample objects' preferences for data recommended in online recommendation scenarios, etc.), the object model can possess extremely strong representational capabilities for objects.

[0172] In summary, in this application, the three training stages—adaptive training of the recommendation model, optimization training of the first recommendation model, and optimization training of the second recommendation model—can all be trained using the aforementioned multimedia resource set and interactive data of several sample objects targeting the multimedia resources in that set. Furthermore, for each training stage, the model can undergo multiple rounds of iterative training. Each round of iterative training can utilize a subset of the multimedia resources in the set and a subset of the sample objects. Within a single training stage, the multimedia resources and sample objects used in different rounds of model training can be different.

[0173] Furthermore, in any training phase and any round of model training, the model parameters can be corrected by the feature similarity between the features generated by each multimedia resource in that round and the features generated by the sample objects in that round. The specific principle of this correction can be found in the principle shown in formula (1) above. For example, in the adaptive training phase of the recommendation model, the model parameters can be corrected by the first feature similarity between the multimedia content features generated by the multimedia resources in the multimedia resource set and the first interaction features generated by the sample objects. Similarly, in the optimization training phase of the first recommendation model, the model parameters can be corrected by the second feature similarity between the multimedia fusion features generated by the multimedia resources in the multimedia resource set and the second interaction features generated by the sample objects. Furthermore, in the optimization training phase of the second recommendation model, the model parameters can be corrected by the third feature similarity between the multimedia fusion features generated by the multimedia resources in the multimedia resource set and the object fusion features generated by the sample objects.

[0174] Figure 11 is a schematic diagram of a process for recommending multimedia resources to an object according to an embodiment of this application. In this embodiment, the trained recommendation model obtained when the aforementioned collaborative features simultaneously include multimedia collaborative features and object collaborative features is used as an example to recall multimedia resources recommended to the object. As shown in Figure 11, the process may include the following steps.

[0175] Step S401: Obtain Q candidate multimedia resources and the target object for resource recommendation, where Q is a positive integer.

[0176] For example, the processing device can acquire Q candidate multimedia resources and a target object for resource recommendation. Q is a positive integer. The Q candidate multimedia resources can include all multimedia resources to be recommended to the target object (i.e., the user). The target object can be the user who needs multimedia resource recommendations, and the number of target objects can be determined according to the actual application scenario. Since the principle of recommending multimedia resources to each target object is the same, the following explanation uses the process of recommending multimedia resources to a single target object as an example. For example, the Q candidate multimedia resources may overlap (i.e., be the same) with the multimedia resources in the aforementioned multimedia resource set, or they may not overlap, depending on the actual application scenario.

[0177] Step S402: Invoke the multimedia model in the trained recommendation model to generate multimedia fusion features for each candidate multimedia resource, and invoke the object model in the trained recommendation model to generate object fusion features for the target object.

[0178] For example, the processing device can invoke the multimedia model in the trained recommendation model to generate multimedia fusion features for each candidate multimedia resource. The processing device can obtain the content data and multimedia collaborative features of the candidate multimedia resources. Therefore, here, the processing device can invoke the multimedia model in the trained recommendation model to generate multimedia fusion features for the candidate multimedia resources using the content data and multimedia collaborative features. A candidate multimedia resource can have one multimedia fusion feature, and a multimedia fusion feature can be a feature vector. The method for obtaining the content data and multimedia collaborative features of the candidate multimedia resources can be the same as the method for obtaining the content data and multimedia collaborative features of the multimedia resources in the multimedia resource set mentioned above.

[0179] For example, the processing device can acquire the content data (referred to as candidate content data) of any candidate multimedia resource, as well as the multimedia collaborative features (referred to as candidate multimedia collaborative features) of that candidate multimedia resource. These candidate multimedia collaborative features can be used to represent the content of that candidate multimedia resource in an online recommendation scenario. Therefore, the processing device can invoke the multimedia model in the trained recommendation model to generate the multimedia fusion features of that candidate multimedia resource using the candidate content data and the candidate multimedia collaborative features.

[0180] The principle of generating multimedia fusion features of candidate multimedia resources by calling the multimedia model in the trained recommendation model through the content data and multimedia collaborative features of candidate multimedia resources is the same as the principle of generating multimedia fusion features of multimedia resources in the multimedia resource set by calling the multimedia model in the first recommendation model in step S203 above through the content data and multimedia collaborative features of multimedia resources in the multimedia resource set (such as the first feature sequence obtained through the content data and multimedia collaborative features). It will not be elaborated here.

[0181] The processing device can also invoke the object model in the trained recommendation model to generate object fusion features of the target object. Similarly, the processing device can obtain the interaction data of the target object with respect to Q candidate multimedia resources, as well as the object collaboration features of the target object. Therefore, the processing device can invoke the object model in the trained recommendation model to generate object fusion features of the target object through the interaction data and object collaboration features of the target object. The method for obtaining the interaction data and object collaboration features of the target object can be the same as the method for obtaining the interaction data and object collaboration features of the sample objects mentioned above.

[0182] For example, the processing device can acquire the target object's interaction data with Q candidate multimedia resources (referred to as target interaction data) and the target object's object collaboration features (referred to as target object collaboration features). The target interaction data is a sequence of L target multimedia resources with which the target object has interactive behavior. These L target multimedia resources can be multimedia resources from the aforementioned Q candidate multimedia resources. The target object collaboration features can be used to represent the target object's object representation in online recommendation scenarios.

[0183] Therefore, the processing device can first call the multimedia model in the trained recommendation model to generate multimedia fusion features for each target multimedia resource. The principle behind generating the multimedia fusion features of the target multimedia resources is the same as the principle behind calling the multimedia model in the first recommendation model to generate multimedia fusion features for the multimedia resources in the multimedia resource set in step S203 above.

[0184] Therefore, the multimedia fusion features of the L target multimedia resources can be input into the object model of the trained recommendation model. This object model then uses the L multimedia fusion features of the L target multimedia resources and the collaborative features of the target object to generate the object fusion features of the target object. These object fusion features can be a feature vector.

[0185] The principle of generating object fusion features of target objects by calling the object model in the trained recommendation model through the interaction data of target objects (such as L multimedia fusion features generated from L target multimedia resources in the interaction data of target objects) and object collaborative features is the same as the principle of generating object fusion features of sample objects by calling the second recommendation model with newly added prefix model parameters in step S303 above through the interaction data of sample objects (such as L multimedia fusion features generated from L multimedia resources in the interaction data) and object collaborative features. It will not be elaborated here.

[0186] Step S403: Obtain the feature similarity between the object fusion features of the target object and the multimedia fusion features of each candidate multimedia resource.

[0187] For example, the processing device can obtain the feature similarity between the object fusion features of the target object and the multimedia fusion features of each candidate multimedia resource. There can be a feature similarity between the object fusion features of the target object and the multimedia fusion features of a candidate multimedia resource. For example, this feature similarity could be a cosine similarity.

[0188] The greater the feature similarity between the object fusion feature of the target object and the multimedia fusion feature of a candidate multimedia resource, the greater the likelihood that the target object is interested in the candidate multimedia resource. For example, the greater the probability that the target object will perform interactive behavior on the candidate multimedia resource. Conversely, the smaller the feature similarity between the object fusion feature of the target object and the multimedia fusion feature of a candidate multimedia resource, the less likely that the target object is interested in the candidate multimedia resource. For example, the less likely that the target object will perform interactive behavior on the candidate multimedia resource.

[0189] Step S404: Sort the Q candidate multimedia resources according to the descending order of feature similarity between the object fusion features of the target object and the multimedia fusion features of each candidate multimedia resource, to obtain the sorted Q candidate multimedia resources.

[0190] For example, the processing device can sort the Q candidate multimedia resources according to the descending order of feature similarity between the object fusion features of the target object and the multimedia fusion features of each candidate multimedia resource, to obtain a sorted set of Q candidate multimedia resources. Among these sorted Q candidate multimedia resources, those with higher feature similarity between their multimedia fusion features and the object fusion features of the target object are ranked earlier, while those with lower feature similarity are ranked later.

[0191] Step S405: Recommend the top R candidate multimedia resources from the sorted Q candidate multimedia resources to the target object; R is a positive integer and R is less than Q.

[0192] For example, the processing device can select the top R candidate multimedia resources from the Q ranked candidate multimedia resources as the multimedia resources recalled for the target object, and recommend these R candidate multimedia resources to the target object (e.g., push these R candidate multimedia resources to the target object's client). R is a positive integer, and R is less than Q. That is, the topR (R with the highest feature similarity) method can be used to recommend multimedia resources to the target object.

[0193] For example, a similarity threshold can be set. Candidate multimedia resources among the Q candidate multimedia resources whose feature similarity between the multimedia fusion features and the object fusion features of the target object is greater than or equal to the similarity threshold can be used as multimedia resources recalled for the target object, and the recalled multimedia resources can be recommended to the target object.

[0194] If the above-mentioned trained recommendation model is trained by introducing multimedia collaborative features but not object collaborative features, then the principle by which the multimedia model in the trained recommendation model generates the features of candidate multimedia resources is the same as the principle by which the multimedia resource features (such as multimedia fusion features) are generated in the embodiment corresponding to Figure 5 above. The principle by which the object model in the trained recommendation model generates the features of the target object is the same as the principle by which the sample object features (such as the first interaction features mentioned above) are generated during the adaptive training of the recommendation model.

[0195] If the above-mentioned trained recommendation model is trained by introducing object collaborative features but not multimedia collaborative features, then the principle by which the object model in the trained recommendation model generates the features of the target object is the same as the principle by which the sample object features (such as object fusion features) are generated in the embodiment corresponding to Figure 8. The principle by which the multimedia model in the trained recommendation model generates the features of multimedia resources is the same as the principle by which the features of multimedia resources (such as the multimedia content features) are generated in the process of adaptively training the recommendation model.

[0196] This application can recall multimedia resources recommended to the target object by using the feature similarity between the features generated by each candidate multimedia resource and the target object after training the recommendation model. For example, it can recall candidate multimedia resources with feature similarity in the top R and recommend them to the target object.

[0197] Figure 12 is a schematic diagram of a scenario for recommending multimedia resources to a user according to an embodiment of this application. As shown in Figure 12, this application can have two application stages: an offline application stage and an online application stage.

[0198] During this offline phase, the processing device can invoke the trained recommendation model to generate user representations (such as object fusion features) for each candidate user (e.g., target object), and can also invoke the trained recommendation model to generate article representations (such as multimedia fusion features) for each candidate multimedia resource (which could be a candidate article). The processing device can store the user representations of each candidate user and the article representations of each candidate multimedia resource in memory (e.g., an in-memory database). The candidate users and candidate multimedia resources can be continuously updated.

[0199] In the subsequent online phase, when a requesting user (which can be any candidate user) needs to view recommended multimedia resources, the processing device can retrieve the user's user representation and the article representations of each candidate article from memory. The processing device can calculate the feature similarity between the user's user representation and the article representations of each candidate article, and can select the top R (i.e., the R articles with the highest feature similarity) candidate articles to recommend to the requesting user.

[0200] Through the above process, namely the excellent trained recommendation model obtained in this application, accurate recommendation of multimedia resources for the target is achieved.

[0201] To verify the accuracy and effectiveness of the recommendation model trained using this application for multimedia resource recall, the following tests were conducted. These tests demonstrate the effectiveness of the model training method and the accuracy of multimedia resource recall.

[0202] Test 1:

[0203] Table 1

[0204] Wherein, Base represents the first recommendation model obtained after pre-training the aforementioned recommendation model in this application; ICF represents model training incorporating multimedia collaborative features (corresponding to training with multimedia collaborative features included, or training with multimedia collaborative features but without object collaborative features); UCF represents model training incorporating object collaborative features (corresponding to the case where collaborative features include object collaborative features, or training with object collaborative features but without multimedia collaborative features); and ICF+UCF represents model training incorporating both multimedia collaborative features and object collaborative features (corresponding to the case where collaborative features simultaneously include multimedia collaborative features and object collaborative features). AUC is a classification evaluation metric, with a higher value indicating better recommendation performance; ACC is an evaluation metric for recommendation accuracy, with a higher value indicating more accurate recommendations.

[0205] Therefore, as shown in Table 1 above, when using the method of this application, the AUC and ACC values ​​for model training with the introduction of multimedia collaborative features are greater than those for adaptive training of the recommendation model alone; the AUC and ACC values ​​for model training with the introduction of object collaborative features are greater than those for model training with the introduction of multimedia collaborative features; and the AUC and ACC values ​​for model training with the simultaneous introduction of multimedia collaborative features and object collaborative features are greater than those for model training with the introduction of object collaborative features alone. This indicates that the simultaneous introduction of multimedia collaborative features and object collaborative features in model training in this application achieves the best effect and accuracy in multimedia resource recommendation.

[0206] Test 2:

[0207] Table 2

[0208] The multimedia resources can be articles. This application conducted five recall tests on articles in the subscription account, each using seven days of data. Recall 50 represents the recall rate when recalling 50 articles, Recall 100 represents the recall rate when recalling 100 articles, and Recall 150 represents the recall rate when recalling 150 articles. As shown in Table 2 above, compared to the Base model (i.e., the first recommendation model after adaptive training of the recommendation model in this application), the recommendation model trained using the multimedia collaborative features and object collaborative features introduced in this application has the highest recall rates in Recall 50, Recall 100, and Recall 150. This clearly demonstrates the effectiveness of introducing multimedia collaborative features and object collaborative features for model training in this application.

[0209] In summary, the test results above definitively confirm the accuracy and effectiveness of the method used in this application for recalling and recommending multimedia resources.

[0210] Figure 13 is a schematic diagram of a model processing device according to an embodiment of this application. As shown in Figure 13, the model processing device 1 may include: a first acquisition module 11, a training module 12, a second acquisition module 13, and an optimization module 14.

[0211] The first acquisition module 11 is used to acquire the content data of multimedia resources in the multimedia resource set and the interaction data of the sample object with respect to the multimedia resources in the multimedia resource set.

[0212] Training module 12 is used to adaptively train the recommendation model using content data and interaction data to obtain the first recommendation model;

[0213] The second acquisition module 13 is used to acquire collaborative features. The collaborative features include at least one of the following: multimedia collaborative features of multimedia resources in the multimedia resource set, and object collaborative features of sample objects, wherein the multimedia collaborative features are used to represent the content representation of multimedia resources in the multimedia resource set in the online recommendation scenario, and the object collaborative features are used to represent the object representation of sample objects in the online recommendation scenario;

[0214] The optimization module 14 is used to optimize and train the first recommendation model using collaborative features to obtain the trained recommendation model; the trained recommendation model is used to recall multimedia resources recommended to the target.

[0215] For example, if the collaborative features include multimedia collaborative features and object collaborative features, then the optimization module 14 uses the collaborative features to optimize and train the first recommendation model to obtain the trained recommendation model in the following ways:

[0216] The first recommendation model is optimized and trained using multimedia collaborative features to obtain the second recommendation model;

[0217] The second recommendation model is optimized and trained using object collaborative features to obtain the trained recommendation model.

[0218] For example, recommendation models include multimedia models and object models. Multimedia models are used to represent multimedia resources, while object models are used to represent objects.

[0219] The adaptive training of the recommendation model includes: simultaneously correcting the model parameters of the multimedia model and the object model in the recommendation model;

[0220] The optimization training of the first recommendation model includes: simultaneously correcting the model parameters of the multimedia model and the object model in the first recommendation model;

[0221] The optimization training of the second recommendation model includes: keeping the model parameters of the multimedia model in the second recommendation model unchanged, and correcting the model parameters of the object model in the second recommendation model.

[0222] For example, the content data is a sequence of K sub-content data belonging to the multimedia resource, where K is a positive integer; the optimization module 14 uses multimedia collaborative features to optimize and train the first recommendation model to obtain the second recommendation model, including:

[0223] The multimedia model in the first recommendation model is invoked to learn features for each sub-content data in the content data, generating sub-content features for each sub-content data.

[0224] The multimedia collaborative features and the K sub-content features of the K sub-content data are sequentially concatenated to obtain the first feature sequence;

[0225] The first recommendation model is optimized and trained based on the first feature sequence to obtain the second recommendation model;

[0226] Among them, there is one multimedia collaborative feature, and the multimedia collaborative feature and K sub-content features are K+1 elements in the first feature sequence. One element is located at the position of an element in the first feature sequence, and the multimedia collaborative feature is located at the position of the first element in the first feature sequence.

[0227] For example, the optimization module 14 optimizes and trains the first recommendation model based on the first feature sequence to obtain the second recommendation model in the following ways:

[0228] The multimedia model in the first recommendation model is invoked to generate multimedia fusion features of multimedia resources in the multimedia resource set based on the first feature sequence;

[0229] The object model in the first recommendation model is invoked to learn features from the interaction data, generating the second interaction features of the sample objects.

[0230] Obtain the second feature similarity between the multimedia fusion features and the second interaction features of the multimedia resources in the multimedia resource set;

[0231] The model parameters of the first recommendation model are corrected based on the similarity of the second feature to obtain the second recommendation model.

[0232] For example, the multimedia model in the first recommendation model has a first attention mask; the optimization module 14 calls the multimedia model in the first recommendation model to generate multimedia fusion features of multimedia resources in the multimedia resource set based on the first feature sequence, including:

[0233] The multimedia model in the first recommendation model is invoked to perform attention processing on the elements in the first feature sequence based on the first attention mask, generating the fused sub-feature corresponding to each element position in the first feature sequence;

[0234] The fusion sub-feature corresponding to the position of the last element in the first feature sequence is used as the multimedia fusion feature of the multimedia resources in the multimedia resource set.

[0235] The first attention mask is used to generate a fusion sub-feature corresponding to the i-th element position based on the element at the i-th element position in the first feature sequence and the elements at the element positions before the i-th element position, where i is a positive integer and i is less than or equal to K+1.

[0236] For example, the interaction data is a sequence of L multimedia resources with interactive behaviors of the sample objects, where L is a positive integer; the optimization module 14 uses object collaborative features to optimize and train the second recommendation model, and the method to obtain the trained recommendation model includes:

[0237] Obtain the multimedia fusion features of each of the L multimedia resources. The L multimedia fusion features of the L multimedia resources are generated by calling the multimedia model in the second recommendation model.

[0238] The object collaborative features are concatenated with the L multimedia fusion features in the form of a prefix of the L multimedia fusion features to obtain the second feature sequence;

[0239] In the object model of the second recommendation model, add L prefix model parameters corresponding to multimedia fusion features, and correct the prefix model parameters based on the second feature sequence to obtain the trained recommendation model;

[0240] Among them, there are P object collaboration features, where P is a positive integer. The P object collaboration features and L multimedia fusion features are the P+L elements in the second feature sequence. One element is located at the position of one element in the second feature sequence. The P object collaboration features are located at the positions of the first P elements in the second feature sequence.

[0241] For example, the optimization module 14 corrects the prefix model parameters based on the second feature sequence to obtain the trained recommendation model in the following ways:

[0242] The multimedia model in the second recommendation model is invoked to generate multimedia fusion features of multimedia resources in the multimedia resource set based on content data and multimedia collaboration features;

[0243] The newly added second recommendation model with prefixed model parameters is invoked to generate object fusion features of the sample objects based on the second feature sequence.

[0244] Obtain the third feature similarity between the multimedia fusion features of multimedia resources in the multimedia resource set and the object fusion features of sample objects;

[0245] The training recommendation model is obtained by correcting the parameters of the newly added prefix model in the second recommendation model based on the third feature similarity.

[0246] For example, the object model in the second recommendation model has a second attention mask; the optimization module 14 calls the newly added second recommendation model with prefixed model parameters to generate object fusion features of the sample objects based on the second feature sequence, including:

[0247] The newly added second recommendation model with prefixed model parameters is invoked. Based on the second attention mask, attention processing is performed on the elements in the second feature sequence to generate the fused sub-feature corresponding to each element position in the second feature sequence.

[0248] The fusion sub-feature corresponding to the position of the last element in the second feature sequence is used as the object fusion feature of the sample object;

[0249] The second attention mask is used to generate a fusion sub-feature corresponding to the j-th element position based on the element at the j-th element position in the second feature sequence and the elements at the element positions before the j-th element position, where j is a positive integer and j is less than or equal to P+L.

[0250] For example, the interaction data is a sequence of L multimedia resources with interactive behaviors of the sample object, where L is a positive integer; the training module 12 uses content data and interaction data to adaptively train the recommendation model to obtain the first recommendation model in the following ways:

[0251] The recommendation model is invoked to learn features from the content data, generating multimedia content features of multimedia resources in the multimedia resource set.

[0252] The recommendation model is invoked to generate multimedia content features of L multimedia resources, and the first interaction features of the sample object are generated based on the L multimedia content features of the L multimedia resources.

[0253] Obtain the first feature similarity between the multimedia content features and the first interaction features of the multimedia resources in the multimedia resource set;

[0254] The model parameters of the recommendation model are corrected based on the similarity of the first feature to obtain the first recommendation model.

[0255] For example, the training module 12 obtains the first recommendation model by correcting the model parameters of the recommendation model based on the first feature similarity in the following ways:

[0256] The interactive behavior information of the sample object to the multimedia resources in the multimedia resource set is obtained. The interactive behavior information is used to indicate whether the sample object has interactive behavior or not interactive behavior to the multimedia resources in the multimedia resource set.

[0257] Based on the interaction behavior information, obtain the reference feature similarity between the sample object and the multimedia resources in the multimedia resource set;

[0258] By analyzing the difference between the first feature similarity and the reference feature similarity, the model parameters of the recommendation model are corrected to obtain the first recommendation model.

[0259] For example, the trained recommendation model includes a multimedia model and an object model, wherein the multimedia model is used to represent multimedia resources and the object model is used to represent objects; the aforementioned model processing device further includes a recommendation module 15, which is used for:

[0260] Get Q candidate multimedia resources and the target object to be recommended, where Q is a positive integer;

[0261] The multimedia model in the trained recommendation model is called to generate the multimedia fusion features of each candidate multimedia resource, and the object model in the trained recommendation model is called to generate the object fusion features of the target object.

[0262] Obtain the feature similarity between the object fusion features of the target object and the multimedia fusion features of each candidate multimedia resource;

[0263] The Q candidate multimedia resources are sorted in descending order of feature similarity between the object fusion features of the target object and the multimedia fusion features of each candidate multimedia resource to obtain the sorted Q candidate multimedia resources.

[0264] The top R candidate multimedia resources from the sorted Q candidate multimedia resources are recommended to the target object; R is a positive integer and R is less than Q.

[0265] For example, the recommendation module 15 calls the multimedia model in the trained recommendation model to generate multimedia fusion features for each candidate multimedia resource in the following ways:

[0266] Obtain candidate content data and candidate multimedia collaborative features for any candidate multimedia resource; the candidate multimedia collaborative features are used to represent the content representation of any candidate multimedia resource in an online recommendation scenario.

[0267] The multimedia model in the trained recommendation model is invoked, and multimedia fusion features of any candidate multimedia resource are generated based on candidate content data and candidate multimedia collaborative features.

[0268] For example, the recommendation module 15 calls the object model in the trained recommendation model to generate object fusion features of the target object in the following ways:

[0269] Obtain the target interaction data and target object collaborative features of the target object; the target interaction data is a sequence of L target multimedia resources with interactive behaviors of the target object, where L is a positive integer; the target object collaborative features are used to represent the object representation of the target object in online recommendation scenarios.

[0270] The multimedia model in the trained recommendation model is invoked to generate multimedia fusion features for each target multimedia resource;

[0271] The object model in the trained recommendation model is invoked, and based on the L multimedia fusion features of L target multimedia resources and the target object collaborative features, the object fusion features of the target objects are generated.

[0272] According to one embodiment of this application, the steps involved in the model processing method shown in FIG3 can be executed by various modules in the model processing device 1 shown in FIG13. For example, step S101 shown in FIG3 can be executed by the first acquisition module 11 in FIG13, step S102 shown in FIG3 can be executed by the training module 12 in FIG13, step S103 shown in FIG3 can be executed by the second acquisition module 13 in FIG13, and step S104 shown in FIG3 can be executed by the optimization module 14 in FIG13.

[0273] According to one embodiment of this application, the modules in the model processing device 1 shown in FIG13 can be individually or entirely merged into one or more units, or some of the units can be further divided into multiple functionally smaller sub-units to achieve the same operation without affecting the technical effect of the embodiment of this application. The above modules are based on logical function division. In practical applications, the function of one module can also be implemented by multiple units, or the function of multiple modules can be implemented by one unit. In other embodiments of this application, the model processing device 1 may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0274] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functions of that module or unit.

[0275] According to one embodiment of this application, a computer program capable of executing the steps involved in the corresponding methods shown in the various embodiments of this application can be run on a general-purpose computer device (which may include processing elements and storage elements such as a central processing unit (CPU), random access storage medium (RAM), and read-only storage medium (ROM)) to construct the model processing apparatus 1 shown in FIG13. The aforementioned computer program can be recorded on a computer-readable recording medium, and can be loaded into and run in the aforementioned computer device via the computer-readable recording medium.

[0276] Please refer to Figure 14, which is a schematic diagram of the structure of a computer device according to an embodiment of this application. As shown in Figure 14, the computer device 1000 may include a processor 1001, a network interface 1004, and a memory 1005. In some embodiments, the computer device 1000 may also include a user interface 1003 and at least one communication bus 1002. The communication bus 1002 is used to implement communication between these components. The user interface 1003 may include a display screen and a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may include, for example, a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM or a non-volatile memory, such as at least one disk storage device. The memory 1005 may also be at least one storage device located remotely from the aforementioned processor 1001. As shown in Figure 14, the memory 1005, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

[0277] In the computer device 1000 shown in Figure 14, the network interface 1004 enables network communication; the user interface 1003 is mainly used for user input; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0278] Acquire the content data of multimedia resources in the multimedia resource set, and the interaction data of sample objects with respect to the multimedia resources in the multimedia resource set;

[0279] The recommendation model is adaptively trained using content data and interaction data to obtain the first recommendation model;

[0280] Acquire collaborative features; collaborative features include at least one of the following: multimedia collaborative features of multimedia resources in the multimedia resource set, and object collaborative features of sample objects, wherein the multimedia collaborative features are used to represent the content representation of multimedia resources in the online recommendation scenario, and the object collaborative features are used to represent the object representation of sample objects in the online recommendation scenario;

[0281] The first recommendation model is optimized and trained using collaborative features to obtain the trained recommendation model; the trained recommendation model is used to recall multimedia resources recommended to the target.

[0282] It should be understood that the computer device 1000 described in the embodiments of this application can execute the model processing method described in the various embodiments of this application, and can also execute the model processing device 1 described in the embodiment corresponding to FIG13 above, which will not be repeated here. In addition, the beneficial effects of using the same method will not be repeated here.

[0283] Furthermore, it should be noted that this application also provides a computer-readable storage medium storing a computer program. When a processor executes this computer program, it can perform the model processing methods described in the various embodiments of this application; therefore, they will not be repeated here. Additionally, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the embodiments of the computer storage medium involved in this application, please refer to the description of the method embodiments of this application.

[0284] As an example, the aforementioned computer program can be deployed and executed on a single computer device, or deployed and executed on multiple computer devices located in one location, or executed on multiple computer devices distributed across multiple locations and interconnected via a communication network. These multiple computer devices distributed across multiple locations and interconnected via a communication network can form a blockchain network.

[0285] The aforementioned computer-readable storage medium can be an internal storage unit of the computer device, such as a hard drive or memory. It can also be an external storage device, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, or flash card. Furthermore, the computer-readable storage medium can include both internal and external storage units of the computer device. This computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. It can also be used to temporarily store data that has been output or will be output.

[0286] This application discloses a computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the model processing methods described in the embodiments of this application; therefore, these will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the embodiments of the computer-readable storage medium involved in this application, please refer to the description of the method embodiments of this application.

[0287] The terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising," and any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0288] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0289] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0290] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A model processing method, comprising: Acquire the content data of multimedia resources in the multimedia resource set, and the interaction data of sample objects with respect to the multimedia resources in the multimedia resource set; The recommendation model is adaptively trained using the content data and the interaction data to obtain a first recommendation model; Acquire collaborative features; the collaborative features include at least one of the features of correlation between objects, between objects and resources, and between resources extracted from interaction data. The first recommendation model is optimized and trained using the collaborative features to obtain a trained recommendation model; the trained recommendation model is used to determine the multimedia resources to recommend to the object.

2. The method according to claim 1, wherein, The collaborative features include multimedia collaborative features and object collaborative features, wherein the multimedia collaborative features are the correlation features between multimedia resources in the multimedia resource set, and the object collaborative features are the correlation features between sample objects; The step of optimizing and training the first recommendation model using the collaborative features to obtain the trained recommendation model includes: The first recommendation model is optimized and trained using the aforementioned multimedia collaborative features to obtain the second recommendation model; The second recommendation model is optimized and trained using the object collaborative features to obtain the trained recommendation model.

3. The method according to claim 1, wherein, The collaborative features include multimedia collaborative features or object collaborative features, wherein the multimedia collaborative features are the correlation features between multimedia resources in the multimedia resource set, and the object collaborative features are the correlation features between sample objects; The step of optimizing and training the first recommendation model using the collaborative features to obtain the trained recommendation model includes: The first recommendation model is optimized and trained using the multimedia collaborative features to obtain the trained recommendation model.

4. The method according to claim 1, wherein, The collaborative features include multimedia collaborative features or object collaborative features, wherein the multimedia collaborative features are the correlation features between multimedia resources in the multimedia resource set, and the object collaborative features are the correlation features between sample objects; The step of optimizing and training the first recommendation model using the collaborative features to obtain the trained recommendation model includes: The first recommendation model is used as the second recommendation model, and the object collaborative features are used to optimize and train the second recommendation model to obtain the trained recommendation model.

5. The method according to claim 2, wherein, The recommendation model includes a multimedia model and an object model. The multimedia model is used to represent multimedia resources, and the object model is used to represent objects. The adaptive training of the recommendation model includes: simultaneously correcting the model parameters of the multimedia model and the object model in the recommendation model; The optimization training of the first recommendation model includes: simultaneously correcting the model parameters of the multimedia model and the object model in the first recommendation model; The optimization training of the second recommendation model includes: keeping the model parameters of the multimedia model in the second recommendation model unchanged, and correcting the model parameters of the object model in the second recommendation model.

6. The method according to claim 3 or 5, wherein, The content data is a sequence consisting of K sub-content data of the respective multimedia resource, where K is a positive integer; the step of optimizing and training the first recommendation model using the multimedia collaborative features to obtain the second recommendation model includes: The multimedia model in the first recommendation model is invoked to perform feature learning on each sub-content data in the content data, and to generate sub-content features for each sub-content data. The multimedia collaborative features and the K sub-content features of the K sub-content data are sequentially concatenated to obtain the first feature sequence; The first recommendation model is optimized and trained based on the first feature sequence to obtain the second recommendation model; The multimedia collaborative features are m in number, and the multimedia collaborative features and the K sub-content features are K+m elements in the first feature sequence, where m is a positive integer. In the first feature sequence, the K sub-content features are located after the multimedia collaborative features.

7. The method according to claim 6, wherein, The step of optimizing and training the first recommendation model based on the first feature sequence to obtain the second recommendation model includes: The multimedia model in the first recommendation model is invoked to generate multimedia fusion features of multimedia resources in the multimedia resource set based on the first feature sequence; The object model in the first recommendation model is invoked to perform feature learning on the interaction data, thereby generating the second interaction feature of the sample object; Obtain the second feature similarity between the multimedia fusion features of the multimedia resources in the multimedia resource set and the second interaction features; The model parameters of the first recommendation model are corrected based on the second feature similarity to obtain the second recommendation model.

8. The method according to claim 7, wherein, The multimedia model in the first recommendation model has a first attention mask; the step of calling the multimedia model in the first recommendation model to generate multimedia fusion features of the multimedia resources in the multimedia resource set based on the first feature sequence includes: The multimedia model in the first recommendation model is invoked to perform attention processing on the elements in the first feature sequence based on the first attention mask, generating fused sub-features corresponding to the position of each element in the first feature sequence. The fusion sub-feature corresponding to the position of the last element in the first feature sequence is used as the multimedia fusion feature of the multimedia resources in the multimedia resource set. The first attention mask is used to generate a fusion sub-feature corresponding to the i-th element position based on the element at the i-th element position in the first feature sequence and the elements at the element positions before the i-th element position, where i is a positive integer and i is less than or equal to K+1.

9. The method according to claim 4 or 5, wherein, The interactive data is a sequence of L multimedia resources with interactive behaviors of the sample object, where L is a positive integer; the step of optimizing and training the second recommendation model using the object collaborative features to obtain the trained recommendation model includes: Obtain the multimedia fusion features of each of the L multimedia resources, wherein the L multimedia fusion features of the L multimedia resources are generated by calling the multimedia model in the second recommendation model; The object collaborative features are concatenated with the L multimedia fusion features in the form of a prefix to obtain the second feature sequence; In the object model of the second recommendation model, add prefix model parameters corresponding to the L multimedia fusion features, and correct the prefix model parameters based on the second feature sequence to obtain the trained recommendation model; There are P object collaboration features, where P is a positive integer. The P object collaboration features and the L multimedia fusion features are P+L elements in the second feature sequence. One element is located at one element position in the second feature sequence. The P object collaboration features are located at the first P element positions in the second feature sequence.

10. The method according to claim 9, wherein, The step of correcting the prefix model parameters based on the second feature sequence to obtain the trained recommendation model includes: The multimedia model in the second recommendation model is invoked to generate multimedia fusion features of multimedia resources in the multimedia resource set based on the content data and the multimedia collaborative features; The second recommendation model, which has been newly added with the prefix model parameters, is invoked to generate object fusion features of the sample object based on the second feature sequence; Obtain the third feature similarity between the multimedia fusion features of the multimedia resources in the multimedia resource set and the object fusion features of the sample objects; The newly added prefix model parameters in the second recommendation model are corrected based on the third feature similarity to obtain the trained recommendation model.

11. The method according to claim 10, wherein, The object model in the second recommendation model has a second attention mask; the invocation of the second recommendation model with the newly added prefix model parameters generates object fusion features of the sample object based on the second feature sequence, including: The second recommendation model with the added prefix model parameters is invoked to perform attention processing on the elements in the second feature sequence based on the second attention mask, generating fused sub-features corresponding to the position of each element in the second feature sequence. The fusion sub-feature corresponding to the position of the last element in the second feature sequence is used as the object fusion feature of the sample object; The second attention mask is used to generate a fusion sub-feature corresponding to the j-th element position based on the element at the j-th element position in the second feature sequence and the elements at the element positions before the j-th element position, where j is a positive integer and j is less than or equal to P+L.

12. The method according to claim 1, wherein, The interaction data is a sequence of L multimedia resources with interactive behaviors of the sample object, where L is a positive integer; the step of adaptively training the recommendation model using the content data and the interaction data to obtain a first recommendation model includes: The recommendation model is invoked to perform feature learning on the content data, generating multimedia content features of multimedia resources in the multimedia resource set; The recommendation model is invoked to generate multimedia content features of the L multimedia resources, and the first interaction features of the sample object are generated based on the L multimedia content features of the L multimedia resources. Obtain the first feature similarity between the multimedia content features of the multimedia resources in the multimedia resource set and the first interaction feature; The model parameters of the recommendation model are corrected based on the first feature similarity to obtain the first recommendation model.

13. The method according to claim 12, wherein, The step of correcting the model parameters of the recommendation model based on the first feature similarity to obtain the first recommendation model includes: The interaction behavior information of the sample object with respect to the multimedia resources in the multimedia resource set is obtained. The interaction behavior information is used to indicate whether the sample object has interactive behavior or not with the multimedia resources in the multimedia resource set. Based on the interaction behavior information, obtain the reference feature similarity between the sample object and the multimedia resources in the multimedia resource set; The model parameters of the recommendation model are corrected by the difference between the first feature similarity and the reference feature similarity to obtain the first recommendation model.

14. The method according to claim 1, wherein, The trained recommendation model includes a multimedia model and an object model, wherein the multimedia model is used to represent multimedia resources, and the object model is used to represent objects; the method further includes: Get Q candidate multimedia resources and the target object to be recommended, where Q is a positive integer; The multimedia model in the trained recommendation model is invoked to generate multimedia fusion features for each candidate multimedia resource, and the object model in the trained recommendation model is invoked to generate object fusion features for the target object. Obtain the feature similarity between the object fusion features of the target object and the multimedia fusion features of each candidate multimedia resource; The Q candidate multimedia resources are sorted in descending order of feature similarity between the object fusion features of the target object and the multimedia fusion features of each candidate multimedia resource to obtain the sorted Q candidate multimedia resources. The top R candidate multimedia resources from the sorted Q candidate multimedia resources are recommended to the target object; R is a positive integer and R is less than Q.

15. The method according to claim 14, wherein, The step of calling the multimedia model in the trained recommendation model to generate multimedia fusion features for each candidate multimedia resource includes: Acquire candidate content data and candidate multimedia collaborative features of any candidate multimedia resource; the candidate multimedia collaborative features are used to represent the content representation of any candidate multimedia resource in an online recommendation scenario. The multimedia model in the trained recommendation model is invoked, and multimedia fusion features of any candidate multimedia resource are generated based on the candidate content data and the candidate multimedia collaborative features.

16. The method of claim 14, wherein, The step of generating the object fusion feature of the target object by calling the object model in the trained recommendation model includes: The target interaction data and the target object collaborative features of the target object are obtained; the target interaction data is a sequence of L target multimedia resources with interactive behavior of the target object, where L is a positive integer; the target object collaborative features are used to represent the object representation of the target object in online recommendation scenarios. The multimedia model in the trained recommendation model is invoked to generate multimedia fusion features for each target multimedia resource; The object model in the trained recommendation model is invoked, and based on the L multimedia fusion features of the L target multimedia resources and the target object collaborative features, the object fusion features of the target object are generated.

17. A model processing apparatus, comprising: The first acquisition module is used to acquire the content data of multimedia resources in the multimedia resource set and the interaction data of the sample object with respect to the multimedia resources in the multimedia resource set. The training module is used to adaptively train the recommendation model using the content data and the interaction data to obtain a first recommendation model; The second acquisition module is used to acquire collaborative features; the collaborative features include at least one of the features of correlation between objects, between objects and resources, and between resources extracted from the interaction data. An optimization module is used to optimize and train the first recommendation model using the collaborative features to obtain a trained recommendation model; the trained recommendation model is used to determine multimedia resources to recommend to an object.

18. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-16.

19. A computer device comprising a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method according to any one of claims 1-16.

20. A computer-readable storage medium storing a computer program adapted to be loaded by a processor and to perform the steps of the method of any one of claims 1-16.

Images