Query Relevance Model Training via Interaction Frequency Filtering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing online concierge systems face accuracy issues in training machine-learned models due to noise in training data caused by user interactions with items unrelated to the query, leading to decreased model performance in determining user interaction probabilities.
Innovation Solution
The online concierge system generates a subset of training data by applying filters based on interaction frequencies, ranks combinations of queries and items, and replicates examples with high interaction frequencies to create a more specific and relevant dataset for model training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If training data is generated from all user interactions with items, then the quantity of training data is increased, but the accuracy of the machine-learned model decreases due to noise from irrelevant interactions
Solution Approach 1:
The training data is segmented into two distinct subsets: a first subset containing high-quality relevant interactions (query-item pairs where the item is actually relevant to the query) and a second subset containing noisy irrelevant interactions. This segmentation allows the model to learn from both data sources while maintaining accuracy by giving appropriate weight to each subset during training.
Solution Approach 2:
The patent modifies the training approach by changing parameters such as using interaction frequency thresholds to filter data, applying different weighting schemes to different data subsets, and adjusting the loss function to account for noise. These parameter changes enable the system to utilize large quantities of data while maintaining model accuracy.
2Measurement precision
If filters based on interaction frequencies are applied to generate a subset of training data, then the accuracy of the model is improved, but the complexity of the training process increases
Solution Approach 1:
The patent applies filtering based on interaction frequencies as a preliminary action before model training. By pre-processing the data to identify and remove low-frequency noisy interactions, the system simplifies the training process while improving accuracy. This preliminary filtering step reduces the complexity of training by providing cleaner input data.
Solution Approach 2:
The system uses the interaction frequency metric itself to automatically filter the training data without requiring complex external validation. The data essentially serves itself by using its own statistical properties (frequency of occurrence) to identify and eliminate noisy examples, reducing the need for manual curation or complex filtering algorithms.
3Reliability
If examples with high interaction frequencies are replicated in the training data, then the relevance of training data to queries is improved, but the quantity of training data increases
Solution Approach 1:
The patent applies partial replication by selectively copying only those training examples that meet a minimum interaction frequency threshold. Rather than replicating all examples equally, the system performs partial action on the most relevant data points, enhancing training quality without proportionally increasing data quantity. This selective approach maintains relevance while controlling data volume.
Data Source
AI summary
An online concierge system maintains various items and an item embedding for each item. When the online concierge system receives a query for retrieving one or more items, the online concierge system generates an embedding for the query. The online concierge system trains a machine-learned model to determine a measure of relevance of an embedding for a query to item embeddings by generating training data of examples including queries and items with which users performed a specific interaction. The online concierge system generates a subset of the training data including examples satisfying one or more criteria and further trains the machine-learned model by application to the examples of the subset of the training data and stores parameters resulting from the further training as parameters of the machine-learned model.