Dynamic Time Windows for Application State Experiments
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Online systems face challenges in controlling for differences between time periods when testing different versions of features or algorithms, leading to inconsistent user interaction data and reduced data collection efficiency.
Innovation Solution
An online system dynamically defines time periods for collecting interaction data based on data collection criteria, incorporating transition periods to ensure consistent testing conditions and accurate data labeling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If versions are tested during different preset time periods, then each version can be tested separately, but differences between time periods (weather, day of week) cannot be controlled for, leading to inconsistent user interaction data
Solution Approach 1:
The patent implements dynamic time period assignment where the duration each version is presented is not fixed but adjusted based on real-time data collection progress. The system monitors interaction data accumulation and dynamically determines when to transition between versions, allowing flexible adaptation to achieve sufficient data collection while maintaining comparable conditions through controlled transition timing.
2Reliability
If versions are frequently changed to test in more similar conditions, then testing conditions become more consistent, but data collected during transition periods cannot be used, reducing data collection efficiency
Solution Approach 1:
The system performs preliminary actions by implementing transition periods between version presentations during which no data is collected. This preparatory transition ensures users fully adapt to the new version before data collection begins, eliminating contamination from previous versions. The transition period is built into the experiment design beforehand to ensure clean data boundaries while maintaining condition similarity.
Solution Approach 2:
The patent extracts and separates the transition period from the data collection period. By explicitly removing the transition phase from usable data collection windows, the system ensures that only clean, uncontaminated data is collected during stable version presentations, while still maintaining frequent version changes for condition similarity.
3Device complexity
If fixed time periods are used for each version, then experiment structure is simple, but cannot adapt to varying data collection rates, leading to inefficient data accumulation
Solution Approach 1:
The system transitions from static fixed time period assignment to dynamic time period assignment. Each version's presentation duration is determined dynamically based on whether sufficient interaction data has been collected, rather than using predetermined fixed intervals. This maintains relatively simple experiment structure while significantly improving data collection efficiency through adaptive timing.
Data Source
AI summary
An online system dynamically determines time periods during which interaction data points are collected for application states being tested as part of an application state experiment. The online system collects interaction data points that occurred after a time when instructions were transmitted to apply the first application state and labels those with a state label corresponding to the first application state. When the online system detects that an interaction data minimum has been met, the online system transmits instructions to the client devices to present a user interface in accordance with a second application state. The online system applies a transition period between when the second set of instructions are transmitted and when the online system starts labeling interaction data points with a state label for the second application state. After the transition period, the online system labels interaction data points with state labels for the second application state.