Two-Model Labeling for Accurate Delayed Event Prediction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine-learned models face accuracy issues when predicting events that are delayed, such as job hires, due to outdated training data and bias introduced by training on data where the earlier event has occurred, leading to poor representation of the population.
Innovation Solution
The use of two separately trained machine-learned models, a label model and a prediction model, where the label model generates labels for training data with known later event outcomes, and the prediction model predicts likelihoods without bias by focusing on data where the earlier event has occurred, even if the later event has not.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If training data is used where the later event outcome is known (e.g., user has been hired or rejected), then the model can learn from complete information, but bias is introduced because the training set only represents users who experienced the prior event (e.g., applied for the job)
Solution Approach 1:
The training data is segmented into two distinct sets: a first training set containing data items where the later event outcome is known (for training the label model), and a second training set containing data items where the later event outcome is unknown but the prior event status is known (for training the prediction model). This segmentation allows each model to be trained on appropriately biased data without contaminating the other, resolving the contradiction between having complete outcome information and maintaining population representativeness.
Solution Approach 2:
A label model is introduced as an intermediary component that generates synthetic labels for the second training set. This label model is trained on the first training set and then used to predict outcomes for data items in the second training set, creating pseudo-labels that allow the prediction model to be trained on a more representative population without directly using biased outcome data.
2Duration of action of moving object
If training data is used where the event is delayed (e.g., job hire takes weeks or months), then the model captures real-world timing, but the training data becomes outdated and accuracy decreases
Solution Approach 1:
The label model is trained in advance on the first training set containing known outcomes, and then used to generate labels for the second training set before the prediction model is trained. This preliminary labeling action allows the prediction model to learn from data that reflects real-world delayed event timing without suffering from outdated information, as the labels are generated based on the temporal relationships captured in the training data.
Data Source
AI summary
The disclosed embodiments provide a method, apparatus, and system for training and using optimizing down funnel predictions using machine-learned labels. More particularly, rather than using a single machine-learned model to predict whether an event (e.g., whether a user will be hired for a particular job) will occur, two separately trained machine-learned models are used. The first model (called the “label model”) is used to create labels for data items (e.g., user profiles and/or other user information, job listing information, etc.) that are obtained, but where it is not known yet whether the event has occurred. These labels may then be combined with those data items and used to train the second model (called the “prediction model”) to learn how to predict whether the event will occur for a data item passed to it.