Neural Network Ranking Model Data Augmentation for Imbalanced Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing learning to rank systems face challenges in handling imbalanced training data distributions, particularly long-tail distributions, which can lead to unfairness for tail queries due to data scarcity and biased model focus on head queries.
Innovation Solution
A data augmentation and contrastive learning framework (DCLR) that synthesizes new training data using adaptive methods and bilateral branch networks to balance data distribution, incorporating contrastive learning to enhance representation learning and joint optimization for improved performance across all queries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If training data is used as-is without augmentation, then training process is simple, but model performance on tail queries deteriorates due to data scarcity
Solution Approach 1:
The patent applies preliminary action by performing data augmentation before model training. Specifically, it synthesizes training data for tail queries using a generator network before the main training process, ensuring that tail queries have sufficient training samples. This preliminary data preparation resolves the data scarcity issue without requiring complex modifications to the training algorithm itself.
Solution Approach 2:
The patent introduces an intermediary component - a generator network - that creates synthetic training data between the original data collection and model training processes. This generator acts as a mediator that transforms limited tail query data into augmented training samples, bridging the gap between data scarcity and model performance requirements.
2Reliability
If model focuses on head queries to maximize overall accuracy, then global performance improves, but fairness to tail queries deteriorates
Solution Approach 1:
The patent applies local quality by treating head queries and tail queries differently in the training process. It synthesizes data specifically for tail queries using a generator network, while head queries use original training data. This localized approach ensures that each query type receives appropriate training attention, improving fairness to tail queries without compromising overall accuracy.
Solution Approach 2:
The patent changes the data distribution parameter by synthesizing additional samples for tail queries. It modifies the training data composition dynamically, creating more balanced class distribution between head and tail queries. This parameter change enables the model to learn from both query types effectively, achieving both accuracy and fairness.
3Reliability
If synthesized training data is generated for all queries, then data distribution balance improves, but training time and computational cost increase
Solution Approach 1:
The patent applies partial action by generating synthesized training data only for tail queries rather than all queries. It identifies queries with insufficient samples (tail queries) and applies data augmentation selectively to these cases. This partial approach achieves data distribution balance while avoiding the excessive computational cost of augmenting already-sufficient head queries.
Data Source
AI summary
A computer-implemented method for training a neural network based ranking model includes performing a training data augmentation operation on a set of training data to generate a set of synthesized training data, and training a neural network based ranking model using the set of training data and the set of synthesized training data. The set of training data includes, for each of a plurality of queries, respective query-document data and respective relevance judgement data. The query-document data for a query includes data associated with a plurality of query-documents pairs for the query. The relevance judgement data for a query includes one or more sets of user feedback data associated with the query. The set of training data has an imbalanced training data distribution and the set of synthesized training data is arranged for use to reduce training data distribution imbalance of the set of training data.