Ranking-Based Customer Prediction Model Evaluation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for evaluating customer prediction models are non-robust to outliers and do not effectively identify the best prospective customers, as they rely on residual-based measures that can be skewed by companies with large wallets, leading to inaccurate targeting in sales and marketing efforts.
Innovation Solution
A ranking-based evaluation method that sorts customer data by predicted spending, calculates ranking order switches, and transforms these measures into a normalized range to assess model performance, providing confidence intervals and graphical representations for selecting the best models for targeting high-potential customers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If residual-based evaluation measures (squared error loss or absolute loss) are used to evaluate prediction models, then the evaluation has statistical likelihood interpretations and represents the true cost of prediction errors, but the evaluation is non-robust to outliers and can be greatly influenced by companies with very large wallets, leading to inaccurate identification of best prospective customers
Solution Approach 1:
The patent transforms the evaluation approach by changing the parameter being measured from residual-based metrics (squared error, absolute error) to ranking-based metrics (Kolmogorov-Smirnov statistic, area under the ROC curve). This parameter change makes the evaluation robust to outliers while maintaining statistical interpretability. The KS statistic measures the maximum vertical distance between the empirical distribution of predicted and actual values, which is insensitive to extreme values.
Solution Approach 2:
The patent replaces the traditional mechanical residual-based evaluation system with a ranking-based evaluation system. Instead of calculating numerical differences between predicted and actual values (mechanical approach), the system ranks predictions and compares distribution functions (statistical approach). This substitution eliminates the sensitivity to outliers inherent in residual-based methods while preserving the ability to measure predictive performance.
2Measurement precision
If mean squared error or mean absolute error measures are used to evaluate prediction models, then the evaluation represents the true cost of prediction errors, but the evaluation is greatly influenced by a small subset of companies with very large wallets, skewing the results
Solution Approach 1:
The patent extracts the ranking information from the predicted values while discarding the absolute magnitude information that causes skewness. By using the KS statistic and AUC, which depend only on the relative ordering of predictions rather than their absolute values, the method removes the harmful influence of companies with very large wallets while retaining the essential predictive signal.
Solution Approach 2:
The patent inverts the traditional evaluation approach by not directly measuring prediction errors (predicted vs. actual differences) but instead measuring the agreement between the ranking of predictions and the ranking of actual outcomes. This inversion transforms a problem sensitive to scale and outliers into one that is robust to these issues.
3Reliability
If median squared error measures are used to evaluate prediction models, then the evaluation is more robust to skewed distribution, but the evaluation completely ignores the performance of the model on companies with large IT wallet size, which are usually the most important customers
Solution Approach 1:
The patent employs multiple ranking-based evaluation metrics (KS statistic, AUC, and concordance correlation) that serve different functions but all address the same underlying problem. The KS statistic provides robustness to skewed distributions, while the AUC maintains sensitivity to important customers by measuring the overall ability to rank them correctly. This multi-functional approach achieves both robustness and accurate performance measurement.
Data Source
AI summary
A method and system perform ranking-based evaluations for regression models that are often appropriate for marketing tasks and are more robust to outliers than traditional residual-based performance measures. The output provided by the method and system provides visualization that can offer insights about local model performance and outliers. Several models can be compared to each other to identify the “best” model and, therefore, the “best” model data for the particular marketing task.