Weighted Consistent Hash for Stable A/B Test Populations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional content variation testing systems fail to maintain a consistent distribution of users across different variants during experiments, especially when variant weights change or variants are added or removed.
Innovation Solution
A variation testing system that uses a weighted consistent hash function to assign users to variants, ensuring a stable population by reassigning users to the 'winning' variant if initially assigned to a 'losing' variant, and maintaining the 'winning' variant for subsequent requests.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional variation testing is used to test content changes, then experimentation can be conducted, but the distribution of users across different variants becomes inconsistent when changes occur such as changing variant weights, adding variants, or removing variants
Solution Approach 1:
The patent applies parameter changes by using a weighted consistent hash function where the weight parameters can be dynamically adjusted. When variant weights change, the hash function automatically reassigns users based on new weights while maintaining consistent distribution. This resolves the contradiction by allowing experiment configuration changes (improving adaptability) while maintaining stable user distribution through the mathematical properties of the hash function (preserving stability).
Solution Approach 2:
The patent replaces conventional mechanical assignment methods (random assignment or fixed assignment) with a hash-based mathematical system. The weighted consistent hash function provides deterministic yet flexible user-to-variant assignment that automatically adapts to weight changes while maintaining consistent distribution. This substitution resolves the contradiction by eliminating the instability inherent in conventional methods while preserving adaptability through parameter-adjustable hash weights.
2Measurement precision
If users are reassigned to winning variants when experiment changes occur, then experiment accuracy is improved, but additional computational overhead is required for reassignment
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing the weighted consistent hash assignments for users before experiment changes occur. When variant weights change, the system only needs to rehash affected users rather than completely reassigning all users. This resolves the contradiction by maintaining measurement accuracy through proper reassignment while minimizing computation time through selective rehashing of only affected users.
Solution Approach 2:
The weighted consistent hash function is self-adjusting and automatically reassigns users based on current variant weights without requiring complex external control logic. The system serves itself by using the hash function's mathematical properties to automatically maintain accurate user distribution across variants. This resolves the contradiction by achieving accurate measurement through automatic self-adjustment while minimizing computational overhead through the efficiency of hash-based assignment.
Data Source
AI summary
A variation testing system environment for performing variation testing of web pages and applications is disclosed. The variation testing system applies a weighted consistent hash function to user attributes of users to assign the users to a variant of a web page that is undergoing experimentation. The usage of the weighted consistent hash function allows for a stable experimental population.