Online Experiment Randomization Evaluation Using Population Stability Index

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional statistical test methods for assessing participant randomization in online experiments often generate false positive and false negative results, leading to unreliable experiment results and requiring manual intervention, which is inefficient and ineffective.

Innovation Solution

An online experiment system implements a Population Stability Index (PSI) test to evaluate the distribution of experiment participants, using a tuning parameter to assess the ratio of participants in one bucket to another, and generates a randomization evaluation to alert for deviations from the expected distribution, thereby ensuring reliable randomization.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If conventional statistical test methods are used to assess participant randomization, then the evaluation process is simple to implement, but the results generate false positives and false negatives leading to unreliable experiment results

Engineering Contradiction:

Improverandomization validation accuracyVSAvoidfalse positive and false negative rate

Core Design Contradiction:

ReliabilityVSMeasurement precision

Solution Approach 1:

The patent changes the fundamental parameter being measured from simple distribution counts to participant attribute characteristics. The PSI test evaluates whether participant attributes (device type, OS, browser, etc.) are similarly distributed across buckets, rather than just counting participants. This parameter transformation eliminates false positives and false negatives by assessing actual randomization quality rather than mere participant counts.

Inventive Principle:

Principle #35Parameter changes

2Productivity

If conventional statistical test methods are used, then manual intervention is required to verify results, but this increases time consumption and reduces efficiency

Engineering Contradiction:

Improverandomization evaluation efficiencyVSAvoidmanual intervention time

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The PSI test is designed to be automatically computable from experiment data without requiring manual verification. The test calculates population stability index values based on participant attribute distributions and automatically determines whether randomization is adequate, enabling the system to self-validate without human intervention and eliminating time losses associated with manual review.

Inventive Principle:

Principle #25Self-service

3Measurement precision

If the PSI test is implemented to reduce false positives and negatives, then the precision and recall of randomization validation improve, but the complexity of the evaluation method increases

Engineering Contradiction:

Improverandomization detection accuracyVSAvoidevaluation system complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The PSI test evaluates randomization by segmenting participants into attribute categories (device type, operating system, browser, etc.) and assessing distribution across buckets for each segment separately. This segmentation approach maintains measurement precision by examining specific attribute distributions while managing complexity through structured categorization of participant characteristics.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The population stability index serves as an intermediary metric that translates complex attribute distribution data into a single interpretable value. This intermediary measurement simplifies the evaluation process by aggregating multiple attribute assessments into one comprehensive indicator, reducing the perceived complexity while maintaining high detection accuracy.

Inventive Principle:

Principle #24Intermediary (Mediator)

4Reliability

If the system generates randomization evaluation during ongoing experiments, then real-time detection of distribution issues is achieved, but the computational load increases

Engineering Contradiction:

Improvereal-time randomization verificationVSAvoidcomputational resource consumption

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The system performs partial evaluation by monitoring a subset of participant attributes and using sampling techniques to assess randomization quality without processing every single participant record in real-time. This partial action approach enables real-time detection of distribution issues while significantly reducing computational resource consumption compared to complete data analysis.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS20230075026A1Online Experiment Randomization Evaluation

Publication Date: 2023.03.09 EBAY INC

AI summary

An online experiment system is described that generates a randomization evaluation for an online experiment, while the experiment is ongoing, indicating whether a distribution of experiment participants allocated to one or more participant groups satisfies an expected distribution. The online experiment system analyzes one of the experiment groups to obtain an observed distribution of the subset of experiment participants included in the experiment group. The online experiment system then evaluates the observed distribution relative to the expected distribution for the experiment according to a decision criteria of a population stability index test. The decision criteria is influenced by a tuning parameter that represents a ratio of experiment participants included in the observed experiment group to experiment participants included in a different experiment group. Responsive to the randomization evaluation indicating that a current distribution of experiment participants fails to satisfy the expected distribution for the experiment, an alert is output.