Sequential Hypothesis Testing for Digital Marketing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional fixed-horizon hypothesis testing in digital marketing environments is inflexible and inefficient, requiring a predetermined sample size and running until a set number of samples are collected, which can lead to inaccurate results and increased time consumption, while also not allowing for real-time monitoring or adjustment of parameters.
Innovation Solution
Sequential hypothesis testing allows for dynamic adjustment of sample size based on real-time data, enabling the test to conclude as soon as statistical significance is reached, permitting real-time monitoring and flexible execution, and reducing the number of user inputs required.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If fixed-horizon hypothesis testing is used with predetermined sample size, then the test structure is simple and easy to implement, but the testing time is extended and efficiency is reduced
Solution Approach 1:
The patent transforms the static fixed-horizon testing approach into a dynamic sequential testing approach where the sample size is not predetermined but determined dynamically based on real-time data. The testing process continues until statistical significance is achieved, allowing the sample size to adapt to the actual data characteristics and effect sizes observed during testing.
Solution Approach 2:
The patent changes the parameter of sample size from a fixed predetermined value to a variable that is determined based on statistical criteria during the testing process. The sample size parameter is adjusted dynamically based on the observed effect size, variance, and desired statistical power, allowing the testing to be both time-efficient and statistically rigorous.
2Ease of manufacture
If fixed-horizon hypothesis testing is used with predetermined sample size, then the test setup is straightforward, but the testing accuracy is reduced
Solution Approach 1:
The patent introduces feedback mechanisms where real-time data from the A/B test is continuously analyzed to determine whether statistical significance has been achieved. This feedback loop allows the testing process to adapt to actual data patterns, ensuring that the test continues long enough to detect meaningful effects while stopping when sufficient evidence is obtained, thereby improving measurement precision.
Solution Approach 2:
The patent performs preliminary calculations of the required sample size based on expected effect sizes and statistical power requirements before initiating the test. This preliminary action provides a guideline for the minimum sample size needed, but the actual testing continues beyond this point if necessary to achieve statistical significance, ensuring both adequate power and accuracy.
3Device complexity
If fixed-horizon hypothesis testing is used, then the test parameters are easy to define, but the adaptability to observed data is reduced
Solution Approach 1:
The patent makes the testing process dynamic by allowing it to adapt to observed data characteristics. The sequential nature of the test enables continuous monitoring of results and adjustment of the stopping decision based on actual data patterns, effect sizes, and statistical significance levels, providing high adaptability while maintaining relatively simple parameter definitions.
4Productivity
If sequential hypothesis testing is used with dynamic sample size adjustment, then the testing efficiency is improved and time is reduced, but the test complexity increases
Solution Approach 1:
The patent uses feedback from real-time data analysis to determine when to stop the test. The continuous monitoring of statistical significance provides feedback that guides the stopping decision, improving efficiency by avoiding unnecessary data collection while maintaining statistical rigor. The complexity is managed through automated calculations and clear statistical criteria.
Solution Approach 2:
The patent enables the testing system to determine its own stopping point based on pre-specified statistical criteria. The system automatically calculates whether statistical significance has been achieved and makes the stopping decision without requiring external intervention, improving efficiency while keeping the overall framework relatively simple through self-determination based on objective criteria.
Data Source
AI summary
Sample size determination techniques in sequential hypothesis testing in a digital medium environment are described. The sample size may be determined before a test to define a number of samples (e.g., user interactions with digital marketing content) that are likely to be tested as part of the sequential hypothesis testing in order to achieve a result. The sample size may also be determined in real time to define a number of samples that likely remain for testing in order to achieve a result. The sample size may be determined in a variety of ways, such as through simulation, based on a gap between conversion rates for different options being tested, and so on.