Content Recommendation Evaluation Using OTR for Cross-Period Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing recommendation engines face challenges in determining high-quality content recommendations during both offline and online inference periods due to differences in input data types, leading to performance gaps and resource wastage from re-evaluating content recommendations.
Innovation Solution
An end-to-end approach using an on-topic-rate (OTR) score to evaluate content recommendations, which minimizes performance gaps by simulating diverse input types from multiple datasets and modifying the machine learning model accordingly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing recommendation engines use different evaluation methods for offline and online inference periods, then they can handle different input data types, but performance gaps occur and resources are wasted from re-evaluating content recommendations
Solution Approach 1:
The patent applies universality by creating a unified evaluation framework that handles both offline and online inference periods using the same OTR score methodology. The system processes diverse input data types (search queries, user profiles, contextual information) through a single evaluation pipeline, eliminating the need for separate evaluation methods and ensuring consistent performance across different inference periods.
2Ease of operation
If existing recommendation engines re-evaluate content recommendations during online inference, then they can adapt to real-time user needs, but computing resources are wasted
Solution Approach 1:
The patent applies preliminary action by pre-computing the OTR score during the offline inference period when the recommendation engine generates content recommendations. This pre-evaluation allows the system to adapt to real-time user needs during online inference without performing additional computationally expensive evaluations, as the quality assessment is already completed in advance.
3Measurement precision
If recommendation engines evaluate content quality during offline inference, then they can improve recommendation accuracy, but the evaluation methods differ from online inference causing performance gaps
Solution Approach 1:
The patent applies parameter changes by standardizing the evaluation parameters to use the OTR score methodology across both offline and online inference periods. This unified parameter approach ensures that content quality is measured consistently regardless of when the evaluation occurs, eliminating performance gaps while maintaining high evaluation accuracy through the sophisticated OTR scoring mechanism.
Data Source
AI summary
Embodiments of the disclosed technologies are capable of evaluating content recommendations. The embodiments describe creating a prompt using a search query and a content recommendation output by a machine learning model in response to the search query. The embodiments further describe causing a LLM to generate an evaluation of the content recommendation and the search query using the prompt. The evaluation includes a relevance score of the content recommendation and the search query. The embodiments further describe training the machine learning model to generate an updated content recommendation in response to the search query. The training includes using the relevance score of the content recommendation and the search query.