Shared Control Pool for A/B Testing Statistical Significance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional A/B testing systems are limited in the number of experiments they can implement per layer due to the requirement of separate control and variant groups, which restricts the sample size and statistical significance, limiting the ability to conduct multiple experiments simultaneously.
Innovation Solution
Implementing a shared control pool that allows multiple experiments to share a predetermined control group, enabling each experiment to utilize a subset of the shared control pool for comparison with variant elements, thereby increasing the number of experiments that can be conducted on a single layer without compromising statistical significance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If separate control and variant groups are used for each experiment, then statistical significance is maintained, but the number of experiments per layer is limited
Solution Approach 1:
Multiple experiments share a common control group instead of each experiment having its own separate control group. The control group is divided into subsets that can be used across different experiments, allowing more experiments to run simultaneously with sufficient statistical power.
Solution Approach 2:
The control group serves multiple functions by being shared across multiple experiments. A single control group can be used for comparison with variant groups from different experiments, making the control group a universal reference point for multiple testing scenarios.
2Measurement precision
If separate control groups are allocated for each experiment, then experimental accuracy is maintained, but memory requirements increase
Solution Approach 1:
Instead of caching separate control and variant elements for each experiment, the system caches a single control group that serves multiple experiments. Variant elements are cached individually, but the control elements are shared, reducing overall memory requirements.
Solution Approach 2:
The control group is copied and reused across multiple experiments rather than creating separate instances. Each experiment references the same control group data, eliminating redundant storage of control elements while maintaining experimental integrity.
Data Source
AI summary
Systems and methods for providing a shared control group are disclosed. A request for a page is received and each user device is assigned to a shared control pool or one of a plurality of experiments. The shared control pool includes a first percentage of user devices and each of the experiments includes an experimental percentage. For each user device in the shared control pool, a base version of each element is transmitted; for each user device in a first experiment, a first variant element is transmitted; and for each user device assigned to a second experiment a second variant element is transmitted. A record indicative of interactions that occur after transmission is received and the interactions of a first subset of the shared control pool is compared with the devices assigned to the first experiment and a second subset is compared to the devices assigned to the second experiment.