A/B Experiment Data Validity via Allocation Error Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing experimental systems for evaluating changes at network accessible sites, such as websites, require lengthy experiments to achieve statistically significant results, leading to resource wastage and delayed implementation of changes, while also risking false negatives in data validity due to improper randomization of user groups.
Innovation Solution
The implementation of a power equation using short-term data to determine the necessary experiment duration and a mechanism to calculate minimal detectable differences, which allows for immediate experimentation and reduces false negatives by assessing data validity, thereby saving resources and accelerating the pace of innovation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If experiments are run for longer durations to achieve statistically significant results, then measurement precision is improved, but loss of time and productivity deteriorate
Solution Approach 1:
The system performs preliminary analysis of historical data and real-time experiment data to calculate the required sample size and projected experiment duration before the experiment reaches completion. This allows operators to know in advance how long the experiment will take, enabling better resource allocation and avoiding unnecessarily long experiment durations while ensuring statistical significance is achieved.
2Reliability
If experiments are run for longer durations to ensure data validity, then reliability is improved, but loss of time and productivity deteriorate
Solution Approach 1:
The system continuously monitors experiment data and provides feedback regarding data validity and potential false negatives. By analyzing data quality metrics in real-time, the system can determine when sufficient valid data has been collected, allowing experiments to be terminated early when reliability requirements are met, thus improving productivity without sacrificing data validity.
3Measurement precision
If historical data is used to determine experiment length, then measurement precision is improved, but loss of time deteriorates due to delayed experiment start
Solution Approach 1:
The system merges historical data analysis with real-time experiment data collection. Instead of completing historical data analysis before starting the experiment, the system begins the experiment immediately while simultaneously analyzing both historical and emerging real-time data to determine the required duration. This combined approach eliminates the delay caused by sequential processing while maintaining the precision of historical data-informed experiment length determination.
4Measurement precision
If large amounts of data are collected to ensure statistical significance, then measurement precision is improved, but use of energy and computing resources worsen
Solution Approach 1:
The system applies partial action by collecting and analyzing only the necessary amount of data required to achieve statistical significance, rather than continuously collecting excessive data. By calculating the precise sample size needed based on historical data and experiment parameters, the system stops data collection when this threshold is reached, reducing unnecessary processor cycles and energy consumption while maintaining measurement precision.
Data Source
AI summary
Technologies are disclosed for determining validity of data obtained from an A/B experiment, where the experiment evaluates the desirability of a potential change at a website. The experiment is run for a period of time and based upon an expected allocation of users into the A group (control group) and the B group (e.g., treatment group), along with an actual number of users directed into the two groups, it is determined if a minimal detectable error of allocation has been exceeded and, if it has, the data is deemed to be invalid. If not, the data is deemed to be valid.