Bayesian A/B Arm Selection for Arbitrary Metric Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing A/B testing systems limit the number of metrics that can be used to determine a winning variation, often relying on manually characterized subsets or user input, which can lead to suboptimal message optimization and difficulty in modeling non-binary metrics like revenue probabilistically.
Innovation Solution
A computer-implemented method using resampling-based Bayesian nonparametrics, specifically a Dirichlet distribution, to model metric densities, allowing for the determination of a winning arm of A/B testing by constructing utility distributions from historical and new data, without requiring user intervention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If A/B testing systems limit the number of metrics to manually characterized subsets or user-provided information, then the system complexity is reduced, but the ability to optimize messages for important metrics is compromised
Solution Approach 1:
The system automatically determines winning arms by soliciting information from the data itself through Bayesian inference and Dirichlet process mixtures, rather than requiring manual metric selection or user input. The algorithm self-adapts to handle any numerical metric including non-binary metrics like revenue, eliminating the need for users to pre-characterize metrics or provide domain expertise.
2Ease of operation
If users are solicited for information such as expected conversion rate and minimum detectable sample size, then the system can function with limited data, but the cognitive load on users increases and users must answer questions they may not be able to answer
Solution Approach 1:
The system eliminates the need for user input by automatically inferring all necessary parameters from the data itself. The Bayesian framework with Dirichlet process mixtures allows the system to self-determine sample sizes, conversion rates, and other parameters without requiring users to answer complex statistical questions, thereby reducing cognitive load to zero.
Solution Approach 2:
The Bayesian inference framework acts as an intermediary that automatically translates raw data into actionable insights without requiring user interpretation. The system mediates between the raw data and the winning arm determination, eliminating the need for users to directly engage with complex statistical concepts.
3Device complexity
If traditional probabilistic modeling is used for non-binary metrics like revenue, then the model structure is simplified, but the ability to accurately model these metrics is significantly reduced
Solution Approach 1:
The system changes the parameterization of the probabilistic model by using Dirichlet process mixtures, which allow flexible modeling of any numerical distribution without being constrained to standard parametric forms. This enables accurate modeling of non-binary metrics like revenue while maintaining computational tractability through the hierarchical Bayesian framework.
Solution Approach 2:
The system combines multiple probabilistic components (Dirichlet process, Gaussian mixtures, hierarchical Bayesian inference) to create a composite modeling framework that can accurately represent complex non-binary metrics. This composite approach integrates the flexibility of non-parametric methods with the computational efficiency of parametric models.
Data Source
AI summary
Apparatuses, methods, and systems for determining winning arms of electronic testing. One method includes obtaining historical data values related to the A/B test of a user, storing the historical data values, determining a historical weight for the historical data values, receiving new data values from the plurality of computing devices collected based on recipient actions during execution of the A/B, constructing a Dirichlet distribution, inferring corresponding central tendencies of samplings of a metric distribution, wherein each central tendency of the corresponding central tendencies is determined by sampling the Dirichlet distribution, constructing an overall utility distribution for each arms of the A/B test by combining the central tendency of each sampling of the metric distribution with a corresponding sampling of a conversion probability distribution, determining a winning arm of the A/B testing by comparing the overall utility distribution of each arm with each other arm of the A/B test.