Quasi-Experimental Testing for Mobile App Bias Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional A/B testing techniques are not valid for evaluating mobile application features due to selection bias between early and late adopters of new versions, leading to incorrect conclusions about user engagement and behavior.
Innovation Solution
A method and system for quasi-experimental testing that uses model-based matching and statistical models, such as propensity score models and endogenous switching models, to estimate average treatment effects and correct for selection bias by comparing users who adopt new versions with similar non-adopters, thereby reducing bias and improving causal inference.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional A/B testing techniques are used to evaluate mobile application features, then the testing process is simple and fast, but selection bias between early and late adopters leads to incorrect conclusions about user engagement and behavior
Solution Approach 1:
The patent introduces statistical models (propensity score models, endogenous switching models) as intermediaries between the treatment assignment and outcome measurement. These models act as mediators that account for selection bias by modeling the probability of adoption based on observed covariates, thereby enabling accurate causal inference in quasi-experimental settings where randomization is not feasible
Solution Approach 2:
The patent transforms the evaluation approach by changing the parameters used in analysis - moving from simple mean comparisons to model-based estimates that incorporate propensity scores and treatment effect heterogeneity. This parameter transformation allows the same data to be analyzed in a way that accounts for selection bias, improving measurement precision without requiring new data collection
2Measurement precision
If model-based matching and statistical models are used to correct for selection bias, then accuracy of feature evaluation is improved, but the complexity of the testing methodology increases
Solution Approach 1:
The patent applies preliminary action by first estimating propensity scores and identifying potential adopters before conducting the main analysis. This preliminary modeling step creates a corrected dataset or weighting scheme that can then be applied to the outcome analysis, separating the complex statistical work into preparatory and execution phases
Solution Approach 2:
The patent segments the user population into distinct groups (adopters, non-adopters, potential adopters) based on their characteristics and adoption probability. This segmentation allows for separate analysis of each group's behavior patterns and treatment effects, making the complex modeling more manageable and interpretable by breaking it into smaller, focused components
Data Source
AI summary
The disclosed embodiments provide a system for evaluating a performance of a mobile application. During operation, the system obtains, for a statistical model used in a quasi-experimental design, a first predicted outcome produced from a first set of data that is collected from two substantially identical versions of a mobile application. Next, the system uses the first predicted outcome to assess a bias of the statistical model. The system then improves an accuracy of the statistical model by using the assessed bias to normalize a second predicted outcome of the statistical model.