Stratified Sampling for Multivariate Testing Variance Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Multivariate testing for web page variations faces challenges in isolating the variance caused by the test experience due to additional variance from user characteristics, leading to less accurate results.
Innovation Solution
Implementing stratified sampling by identifying stratification factors such as age range and client device type, generating pseudorandom numbers to assign users to variations, and tracking retention information to minimize non-test variance, ensuring each variation is equally represented across user permutations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multivariate testing is performed without stratified sampling, then the testing process is simpler and easier to implement, but the test results have higher variance due to user characteristics, reducing measurement precision
Solution Approach 1:
The patent applies segmentation by dividing the user population into distinct strata based on characteristics such as age range and client device type. This stratification allows the testing process to isolate and control for variance caused by user characteristics, thereby improving measurement precision of test results while managing complexity through structured organization of user groups
Solution Approach 2:
The patent changes the parameter of user assignment from simple random allocation to stratified random allocation based on user characteristics. By introducing stratification parameters (age range, device type), the system transforms the testing approach to reduce variance in test results, improving measurement precision without prohibitively increasing complexity
2Measurement precision
If stratified sampling is implemented to reduce variance, then test result accuracy improves, but the complexity of user assignment and tracking increases
Solution Approach 1:
The patent applies preliminary action by pre-defining strata and establishing pseudorandom number generation schemes before the multivariate test begins. User assignments are determined in advance based on their strata membership, which simplifies the actual testing process and reduces the complexity of real-time user assignment while maintaining improved measurement precision
Solution Approach 2:
The patent uses pseudorandom numbers as a copy or surrogate for complex random assignment processes. By generating pseudorandom assignments based on user strata, the system creates a simplified representation of randomization that reduces computational and operational complexity while preserving the statistical benefits of reduced variance in test results
Data Source
AI summary
Embodiments provide techniques for testing a plurality of variations of a user experience on a subscription-based online site, where each of the plurality of variations is distinct from other variations in the plurality of variations. Each of a plurality of users is assigned to one of the plurality of variations. For each of a plurality of user requests received during a predetermined window of time, embodiments determine one of the plurality of variations for use in processing the user request, based on a user associated with the user request, and process the respective user request using the determined variation. Long-term user metric information relating to subscription services for the subscription-based online site is determined, with respect to the plurality of users. Embodiments then rate at least one of the plurality of variations, based on the determined long-term user metric information.