Offline Simulation for Experiment Budget Adjustments
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Online concierge systems face challenges in evaluating the impact of new features on user interactions due to uneven budget usage across multiple experiments, leading to distorted metrics and inadequate sample sizes, which complicates the evaluation of item prioritization and presentation strategies.
Innovation Solution
An offline simulation method is employed to replay online experimental data, adjusting metrics to account for uneven budget usage by comparing actual budget usage to a fair value, allowing for multiple experiments to be evaluated without affecting live presentation budgets and ensuring adequate sample sizes, while also accounting for campaign-campaign interactions and budget splits.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple experiments are performed on live users with live content campaigns and presentation budgets, then real-world experimental data can be collected, but the presentation budget is split across many experiments limiting the budget available for each variant and creating unacceptably small sample sizes
Solution Approach 1:
The patent creates synthetic copies of presentation budgets and campaign data to simulate multiple experiments. Instead of running experiments on live users with actual budgets, the system generates synthetic experiment data that mirrors real-world conditions, allowing multiple experiments to be evaluated simultaneously without competing for real budget resources.
Solution Approach 2:
The system performs preliminary simulation of experiments using historical and synthetic data before deploying to live users. By pre-evaluating experiment designs and predicting outcomes on synthetic data, the system can identify promising experiments worth running on live users while avoiding those that would waste budget or produce insufficient sample sizes.
2Measurement precision
If multiple experiments are performed on live users with live content campaigns and presentation budgets, then real-world experimental data can be collected, but it is difficult to directly determine how new features affect prioritized items and presentation budget usage
Solution Approach 1:
The patent introduces synthetic presentation budgets as an intermediary layer between the experiment design and live user data. This synthetic budget layer acts as a mediator that isolates the evaluation of feature impacts from the complexity of actual budget allocation across multiple competing experiments, allowing cleaner measurement of how new features affect prioritization.
Solution Approach 2:
The system extracts the budget allocation logic from the live experiment system and implements it separately in the simulation environment. By taking out the budget management complexity from the live system and handling it independently in synthetic experiments, the system can more easily determine feature impacts without the confounding effects of real-world budget constraints.
3Productivity
If variants exhaust the same presentation budget early with uneven usage rates, then experiments can be completed quickly, but metrics become distorted and fair comparison between variants is compromised
Solution Approach 1:
The patent applies counterweight adjustments to balance the uneven budget consumption across variants. When one variant exhausts the budget faster than others, the system applies corrective weighting to the metrics to compensate for this imbalance, ensuring that faster completion does not create distorted comparisons. This counterbalancing allows quick experiment completion while preserving metric comparability.
Data Source
AI summary
An online concierge system may conduct experiments in presentation of prioritized items for content campaigns with offline simulations. The offline simulation may use a joint budget for the content campaign used by several experimental variations that affect prioritized content presentation. To correct for distortions that may occur from differing rates of budget use in the variations when the budget is reached before a total period for the experiment, the budget use of each variation is compared to a “fair value” to determine an adjustment to the metrics determined in the experiment. Variants that exceed the fair value may have their metrics capping to the portion allocable to a budget use that does not exceed the fair value, while variants that use less than the fair value may have the metrics extrapolated to account for the additional budget that would be available with a fair value budget.