Imbalance Detection in Online Experiments via Permutation Tests
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing experimentation methods, such as A/B testing, often face challenges in detecting imbalances in user populations, leading to unreliable and resource-intensive results.
Innovation Solution
The proposed solution involves a resource-efficient technique for automatically detecting imbalances in experimentation by using permutation tests applied to contingency tables, which can handle large data sets and run independently in parallel, leveraging distributed computing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If permutation tests are applied to raw data points directly, then measurement precision improves, but computing resource consumption increases significantly
Solution Approach 1:
The patent extracts only the essential features needed for imbalance detection by applying permutation tests to contingency tables derived from aggregated experiment data, rather than processing all raw data points. This extraction approach maintains detection precision while significantly reducing computational resource consumption by working with summarized statistics instead of individual data points.
2Productivity
If permutation tests are run in parallel across multiple experiments, then productivity improves, but device complexity increases
Solution Approach 1:
The patent segments the overall experimentation process into independent units that can be processed in parallel. Each experiment's contingency table analysis is divided into separate permutation test tasks that can execute concurrently across multiple processing units. This segmentation enables scalable parallel processing that increases productivity while managing device complexity through modular task design.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer-storage media, for imbalance detection in online experiments. In some implementations, a method includes obtaining user information indicating a first set of devices assigned a first version of a service and a second set of devices assigned a second version of the service in a multivariate testing framework; generating alternative samplings of devices; generating a threshold for detecting imbalance using the alternative samplings of devices, a generated expected first set of devices, and a generated expected second set of devices; detecting an imbalance using the obtained user information indicating the first set of devices, the second set of devices, and the generated threshold; and implementing a corrective action to mitigate the detected imbalance in the multivariate testing framework.