Lift Reporting With ML Causal Conversion Ranking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing lift reporting systems fail to capture causal/incremental performance of content, leading content creators to optimize towards performance metrics that are not necessarily the most important, as they rely on attributed conversions.
Innovation Solution
A lift reporting system that provides new reports and metrics based on causal effects of ads, using machine-learned models to analyze user behavior data and determine causal conversions, enabling A/B testing to rank and implement models that optimize for desired user actions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If attributed conversion metrics are used for A/B testing, then campaign optimization can be performed, but the metrics do not capture causal/incremental performance leading to suboptimal decisions
Solution Approach 1:
The system segments users into treatment groups and control groups to isolate causal effects. By dividing the user population and applying different ML models to treatment groups while keeping control groups unchanged, the system can measure incremental conversions that are directly attributable to specific model changes, thereby capturing causal performance information that traditional attributed conversion metrics miss.
Solution Approach 2:
The system introduces an intermediary layer of causal analysis between traditional attribution and optimization decisions. Lift reporting acts as this intermediary, using statistical methods to separate true causal impact from spurious correlations in conversion data, thereby providing more accurate performance metrics for campaign optimization without losing the nuanced causal information.
2Productivity
If traditional A/B testing with attributed conversions is used, then campaign performance can be measured, but causal/incremental performance remains uncaptured
Solution Approach 1:
The system implements feedback loops where lift reporting results feed back into ML model optimization decisions. By continuously measuring causal impact through controlled experiments and using these results to guide model updates, the system creates a closed-loop optimization process that captures causal performance information and uses it to improve campaign productivity over time.
Solution Approach 2:
The system performs preliminary causal analysis through controlled A/B testing before full campaign deployment. By pre-testing ML model changes on treatment groups and measuring incremental conversions before wide rollout, the system captures causal performance information in advance, enabling more informed optimization decisions and improving overall campaign productivity.
3Ease of operation
If content creators optimize based on attributed conversions, then campaign deployment can proceed, but optimization targets may not be the most important metrics
Solution Approach 1:
The system provides feedback to content creators through lift reporting that highlights which metrics truly drive causal performance. By showing the impact of different factors on incremental conversions rather than just attributed conversions, the system guides creators to prioritize the right metrics while maintaining ease of deployment through automated reporting and recommendations.
Data Source
AI summary
A lift reporting system to perform operations that include: accessing user behavior data associated with one or more machine-learned (ML) models, the ML models associated with identifiers; determining causal conversions associated with the ML models based on the user behavior data, the causal conversions comprising values; performing a comparison between the values that represents the causal conversions; determining a ranking of the ML models based on the comparison; and causing display of a graphical user interface (GUI) that includes a display of identifiers associated with ML models.