Stopping Rules
Knowing when to stop an experiment is a critical aspect of trustworthy A/B testing. Stopping rules are the guidelines you set before running your experiment that dictate when you will analyze and potentially end the test. These rules help you avoid drawing invalid conclusions due to random fluctuations in your data.
There are two primary types of stopping rules:
-
Fixed-sample stopping rule: Decide in advance how many samples or how much time you will collect data before looking at the results. For example, you might plan to stop collecting data after 10,000 users have participated or after four weeks, whichever comes first. This approach is straightforward and helps protect against bias introduced by repeatedly checking the results.
-
Sequential approaches: Allow for monitoring the data at multiple points during the experiment. With sequential rules, you might check the results every day or week and have pre-defined criteria for stopping early if the evidence is strong enough. However, sequential monitoring requires careful statistical adjustments to maintain the validity of your conclusions, as frequent peeking increases the risk of false positives.
Peeking at results before your pre-defined stopping pointβor stopping the experiment early based on interim resultsβcan dramatically increase the risk of Type I errors (false positives). This practice, known as "peeking," can lead you to believe you have found a significant effect when it is actually due to random chance. To avoid these risks, always set and adhere to your stopping rules before starting data collection. For further reading, see discussions on alpha spending and sequential analysis in statistical literature.
Thanks for your feedback!
Ask AI
Ask AI
Ask anything or try one of the suggested questions to begin our chat
Awesome!
Completion rate improved to 3.23Stopping Rules
Swipe to show menu
Knowing when to stop an experiment is a critical aspect of trustworthy A/B testing. Stopping rules are the guidelines you set before running your experiment that dictate when you will analyze and potentially end the test. These rules help you avoid drawing invalid conclusions due to random fluctuations in your data.
There are two primary types of stopping rules:
-
Fixed-sample stopping rule: Decide in advance how many samples or how much time you will collect data before looking at the results. For example, you might plan to stop collecting data after 10,000 users have participated or after four weeks, whichever comes first. This approach is straightforward and helps protect against bias introduced by repeatedly checking the results.
-
Sequential approaches: Allow for monitoring the data at multiple points during the experiment. With sequential rules, you might check the results every day or week and have pre-defined criteria for stopping early if the evidence is strong enough. However, sequential monitoring requires careful statistical adjustments to maintain the validity of your conclusions, as frequent peeking increases the risk of false positives.
Peeking at results before your pre-defined stopping pointβor stopping the experiment early based on interim resultsβcan dramatically increase the risk of Type I errors (false positives). This practice, known as "peeking," can lead you to believe you have found a significant effect when it is actually due to random chance. To avoid these risks, always set and adhere to your stopping rules before starting data collection. For further reading, see discussions on alpha spending and sequential analysis in statistical literature.
Thanks for your feedback!
