LLM Encoder Evaluation Model for Content Relevance Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Algorithmic content recommendation systems face challenges in accurately evaluating the relevance of content impressions, particularly due to the difficulty in verifying the irrelevance of predictions and the limited quality of training data, leading to irrelevant recommendations that can frustrate users and diminish platform trust.
Innovation Solution
An evaluation model leveraging a large language model encoder is employed to assess the relevancy of content impressions, using diverse training data and fine-tuning to improve scalability and accuracy, with features like random negative data and Embedding Based Retrieval (EBR) to enhance the quality of labeled negative training data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional predictive models are used for content recommendation, then the system can operate with simpler architecture, but the accuracy and reliability of relevance evaluation deteriorates
Solution Approach 1:
The patent introduces an evaluation model as an intermediary component that independently assesses the predictions made by the predictive model. This evaluation model uses a large language model encoder to generate embeddings and determine relevance, acting as a mediator between the predictive model and the final recommendation output, thereby improving accuracy without completely redesigning the core recommendation architecture
Solution Approach 2:
The system is divided into distinct functional modules: a predictive model for initial recommendations, an evaluation model for accuracy assessment, and a large language model encoder for generating embeddings. This segmentation allows each component to specialize in specific tasks, improving overall relevance evaluation accuracy while maintaining manageable system complexity through modular design
2Measurement precision
If diverse training data is used to improve model accuracy, then the quality of recommendations improves, but the difficulty and cost of data preparation increases
Solution Approach 1:
The system employs self-service mechanisms for data preparation through automated generation of negative training data. The large language model encoder automatically generates embeddings and determines relevance without requiring manual annotation of all training data, significantly reducing the effort and cost of preparing diverse training data while maintaining high recommendation quality
Solution Approach 2:
The patent performs preliminary actions by pre-computing embeddings using the large language model encoder and preparing diverse training data in advance. This includes generating both positive and negative examples beforehand, which simplifies the subsequent model training process and reduces the complexity of data preparation at deployment time
3Measurement precision
If manual verification of prediction relevance is performed, then the accuracy of evaluation improves, but the productivity and scalability of the system deteriorates
Solution Approach 1:
The patent replaces manual verification (mechanical human process) with an automated evaluation model that uses a large language model encoder. This substitution maintains high verification accuracy through sophisticated embedding-based relevance determination while dramatically improving productivity and scalability by eliminating the need for human annotators to review each prediction
4Productivity
If the evaluation model uses large language model encoder, then the scalability of the system improves, but the computational resources and energy consumption increases
Solution Approach 1:
The system applies partial action by using the large language model encoder selectively - primarily for generating embeddings and evaluating predictions rather than processing all possible data. The evaluation model uses these embeddings to determine relevance without requiring the full computational power of the LLM encoder for every operation, thus achieving scalability while managing energy consumption
Data Source
AI summary
Aspects of the disclosure include methods for evaluating a predictive model. An exemplary method includes training an evaluation model to output, for an input first entity-second entity pair, a content relevancy prediction. A large language model encoder of the evaluation model generates a first embedding for the first entity and a second embedding for the second entity. The embeddings are fed to an interaction tower to produce a logit and the logit is passed with true labels to a loss function for fine-tuning. The true labels include labeled training data generated by modifying training data having a first proportion of negative labeled data to provide a second proportion of negative labeled data greater than the first proportion. The evaluation model is used to score a performance of a predictive model based at least in part on a comparison of predictions made by the respective models for a same entity pair.