Cumulative Incrementality Score for ML Model Evaluation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional machine-learning metrics for evaluating incrementality models in online concierge systems are inefficient, as they do not effectively target users likely to perform a target action, leading to excessive resource usage and unnecessary treatments.
Innovation Solution
The online concierge system employs a cumulative incrementality score to evaluate and select incrementality models, ensuring treatments are applied only to users with high predicted likelihood of performing a target action, thereby reducing resource usage and increasing treatment effectiveness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional machine-learning metrics are used to evaluate incrementality models, then the model performance is measured across the full range of predictions, but the system wastes resources treating users who would not perform the target action even if treated
Solution Approach 1:
The patent changes the evaluation parameter from conventional metrics (accuracy, precision, recall) to cumulative incrementality score, which measures only the additional value generated by treatments. This parameter change allows the system to focus evaluation on high-impact users while ignoring low-value cases, thereby reducing computational waste.
Solution Approach 2:
The patent extracts the evaluation focus from the full prediction range and isolates it to the high-end predictions where treatments actually matter. By using cumulative incrementality score, the system separates and evaluates only the portion of predictions that lead to meaningful user actions, discarding unnecessary evaluation of users who would not respond to treatments anyway.
2Ease of operation
If treatments are applied to all users to ensure coverage, then the system maintains simplicity in treatment application, but the rate of user interaction decreases due to excessive resource consumption
Solution Approach 1:
The patent applies partial action by treating only a subset of users (those with high predicted incrementality scores) rather than all users. This selective treatment approach maintains operational simplicity while improving user interaction rates by avoiding unnecessary treatments that would waste resources and potentially annoy users.
Solution Approach 2:
The system uses cumulative incrementality score as feedback to guide treatment allocation. This feedback mechanism allows the system to automatically identify which users are most likely to respond to treatments, creating a closed-loop system that optimizes both operational ease and user interaction rates without manual intervention.
3Reliability
If the system focuses on identifying users who will perform target actions, then treatment effectiveness increases, but the complexity of model evaluation increases
Solution Approach 1:
The patent simplifies the evaluation complexity by changing to a single cumulative incrementality score parameter that directly measures treatment effectiveness. This single metric consolidates multiple evaluation dimensions into one clear indicator, making the system both reliable for identifying effective users and relatively simple to implement and interpret.
Data Source
AI summary
An online concierge system uses a cumulative incrementality score to evaluate the performance of incrementality models used by the online concierge system to identify users for treatment. The online concierge system applies an incrementality model to a set of examples to generate predicted incrementality scores for the examples. The online concierge system ranks the examples based on the predicted incrementality scores for the examples and groups the examples based on their rankings. The online concierge system iteratively computes cumulative incrementality scores for each grouping based on the examples of each grouping, and computes a final cumulative incrementality score for the incrementality model based on each of the cumulative incrementality scores.