Plural-Objective Model Training for Novel, Relevant Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine-trained models for matching source items to target items primarily focus on relevance, often neglecting novelty, leading to repetitive and less engaging results, and lack sufficient user feedback for optimizing novelty.
Innovation Solution
A reinforcement learning-based model is trained to promote both relevance and novelty using a dual-objective approach, incorporating a novelty-reference model and a relevance-reference model to generate rewards, reducing the action space to binary outcomes, and utilizing a dual-encoder architecture for efficient training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a model focuses on relevance using supervised learning, then matching accuracy is improved, but novelty is neglected leading to repetitive results
Solution Approach 1:
The patent changes the training objective from single-criterion relevance optimization to dual-criterion optimization by modifying the reward function to include both relevance and novelty components. This parameter change in the objective function enables the model to simultaneously improve matching accuracy while enhancing novelty through reinforcement learning.
Solution Approach 2:
The patent segments the matching objective into two distinct components: relevance (measured by reference model agreement) and novelty (measured by diversity from reference results). By separating these objectives and optimizing them jointly through a composite reward function, the system achieves both accurate matching and novel results without compromising either criterion.
2Adaptability or versatility
If reinforcement learning with large action space is used, then model flexibility is improved, but convergence rate decreases and resource consumption increases
Solution Approach 1:
The patent extracts and removes the selection decision from the action space, keeping only the binary match/no-match decision. By taking out the complex ranking and selection operations from the reinforcement learning action space, the model achieves faster convergence and lower resource consumption while maintaining flexibility through the reward function design that incorporates both relevance and novelty criteria.
3Measurement precision
If reference models are used to generate rewards, then novelty assessment is improved, but system complexity increases
Solution Approach 1:
The patent introduces reference models as intermediary components that provide guidance signals for reward generation. These reference models (including novelty-reference and relevance-reference models) act as mediators that translate the complex novelty and relevance assessment into computable reward signals, enabling precise measurement without requiring the main model to directly implement complex assessment logic.
Solution Approach 2:
The patent implements feedback mechanisms where reference models evaluate candidate matches and provide reward signals back to the reinforcement learning model. This feedback loop enables continuous improvement of matching performance by comparing model outputs against reference standards for both novelty and relevance, creating a self-correcting system that maintains assessment precision.
4Adaptability or versatility
If a model prioritizes novelty, then result diversity is improved, but relevance may be compromised
Solution Approach 1:
The patent changes the optimization parameters by introducing a composite reward function that balances novelty and relevance with adjustable weighting. By modifying the reward structure to include both diversity metrics (novelty) and quality metrics (relevance), the system can tune the balance between these competing objectives through hyperparameter optimization, ensuring neither criterion is compromised.
Solution Approach 2:
The patent applies local quality optimization by allowing different regions of the search space to be optimized for different criteria. Candidate items that are highly relevant can be optimized for relevance, while items on the boundaries of the result set can be optimized for novelty. This local differentiation enables the system to maintain high relevance for core results while incorporating novel items without sacrificing overall quality.
Data Source
AI summary
A technique uses reinforcement learning to train a plural-objective model that generates target items based on the dual objectives of relevance and novelty. The reinforcement learning expresses each state as a combination of a particular source item (e.g., a query) and a particular target item. The reinforcement learning generates an action that indicates whether the target item is selected as a good match for the source item. The reinforcement learning then generates a reward based on the state and the action. In doing so, the reinforcement learning relies on a novelty-reference model for assessing novelty and a relevance-reference model (e.g., a large language model) for assessing relevance. The reinforcement learning then uses the reward to update parameters of the plural-objective model.