Model-Based Matching for Quasi-Experimental Mobile App Testing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional A/B testing methods are not valid for evaluating mobile application features due to selection bias between early and late adopters of new versions, leading to incorrect conclusions about user engagement and behavior.

Innovation Solution

A method and system for quasi-experimental testing that uses model-based matching and statistical models to estimate average treatment effects, validate bias, and correct for selection bias by comparing adopters and non-adopters, employing propensity score models and endogenous switching models to improve causal inference and reduce bias in user behavior evaluation.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If conventional A/B testing is used to compare user behavior between new and old versions, then the evaluation process is simple, but selection bias between adopters and non-adopters leads to incorrect conclusions

Engineering Contradiction:

Improveevaluation process simplicityVSAvoiduser behavior measurement accuracy

Core Design Contradiction:

Ease of operationVSMeasurement precision

Solution Approach 1:

The patent introduces propensity scores as an intermediary variable that mediates the relationship between user characteristics and version adoption. By calculating propensity scores based on user covariates and using these scores to match or weight users, the method creates a balanced comparison group that eliminates selection bias while maintaining evaluation feasibility

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent transforms the evaluation approach by changing the parameter of user selection from random assignment to propensity-score-based matching. This parameter change allows the construction of comparable groups with similar distributions of covariates, thereby removing selection bias and improving measurement precision

Inventive Principle:

Principle #35Parameter changes

2Measurement precision

If quasi-experimental testing is implemented to correct for selection bias, then measurement precision improves, but the complexity of the evaluation process increases

Engineering Contradiction:

Improveuser behavior measurement accuracyVSAvoidevaluation process complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent replaces the mechanical process of manual user matching and bias adjustment with automated statistical computations. By using propensity score models and automated matching algorithms, the system performs complex bias correction calculations programmatically, reducing the perceived complexity for users while maintaining high measurement precision

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS10372599B2Model-based matching for removing selection bias in quasi-experimental testing of mobile applications

Publication Date: 2019.08.06 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

The disclosed embodiments provide a system for evaluating a performance of a mobile application. During operation, the system obtains a first set of data associated with adopters of a new version of a mobile application in a partial rollout of the new version and a second set of data associated with non-adopters of the new version in the partial rollout. Next, the system applies a statistical model to the first and second sets of data to select a subset of the non-adopters as potential adopters of the new version. The system then reduces a bias in a quasi-experimental design associated with the mobile application by using the first set of data and a third set of data associated with the potential adopters to estimate an average treatment effect (ATE) between the new version and an older version of the mobile application.