Statistical Hypothesis Testing System for Long-Term Effect Determination

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Statistical hypothesis testing, such as A/B testing, often yields initial results that change over time, making it impractical and costly to maintain experiments for extended periods to determine long-term effects of website features, as merchants may implement features based on short-term benefits without knowing their long-term impact.

Innovation Solution

The system determines long-term effects by using data from time periods external to the experiment, including data collected before and after the experiment, allowing for the assessment of feature implementations based on extended user interactions and behaviors.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If statistical hypothesis testing is conducted for extended periods to determine long-term effects, then measurement precision of long-term effects is improved, but loss of time and productivity deteriorate

Engineering Contradiction:

Improvemeasurement precision of long-term effectsVSAvoidloss of time maintaining experiments

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system performs preliminary actions by collecting user interaction data during the experiment period and automatically projecting long-term effects using statistical models before the experiment concludes. This allows long-term效果 assessment without actually maintaining the experiment for the full long-term period, thus resolving the contradiction between measurement precision and time loss.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system creates a computational model (copy) of the experiment that simulates long-term user interactions and effects. Instead of running the actual experiment for extended periods, the model replicates and projects outcomes, achieving long-term measurement precision without the corresponding time investment.

Inventive Principle:

Principle #26Copying

2Reliability

If statistical hypothesis testing is conducted for extended periods to determine long-term effects, then reliability of feature implementation decisions is improved, but productivity deteriorates

Engineering Contradiction:

Improvereliability of feature implementation decisionsVSAvoidproductivity of feature implementation

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The system uses computational models to copy and simulate long-term experiment outcomes, providing reliable decision-making data without actually maintaining experiments for extended periods. This maintains reliability while preserving productivity by avoiding the time-consuming nature of long-term actual experimentation.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The system replaces the mechanical process of actually running long-term experiments with an automated computational modeling system. This substitution maintains the reliability of results while dramatically improving productivity by eliminating the need for prolonged experiment maintenance.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Productivity

If merchants implement features based on short-term benefits, then productivity is improved, but loss of information about long-term impact worsens

Engineering Contradiction:

Improveproductivity of feature implementationVSAvoidloss of information about long-term impact

Core Design Contradiction:

ProductivityVSLoss of information

Solution Approach 1:

The system provides feedback by generating projected long-term effect data that accompanies short-term experiment results. This allows merchants to make informed decisions based on both immediate and anticipated long-term impacts, preventing information loss about long-term consequences while maintaining quick decision-making capability.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The system performs preliminary analysis of potential long-term effects before merchants make implementation decisions. By providing this advance information about long-term impacts, merchants can act quickly with complete information, resolving the contradiction between productivity and information completeness.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS10152458B1Systems for determining long-term effects in statistical hypothesis testing

Publication Date: 2018.12.11 AMAZON TECH INC

AI summary

Described are techniques for determining long-term effects of an experimental change to a user experience after the end of the experiment. A control state and a treatment state of a statistical hypothesis experiment may be assigned to first and second client devices, respectively, during an experiment time period. Subsequent to the end of the experiment, presentation of the control state may be discontinued. Result data corresponding to the treatment state may be determined during the experiment time period and for a length of time subsequent to the experiment time period. Result data corresponding to the control state may be determined during the experiment time period, and for a length of time prior to assignment of the control state to the first client device.