How Not to Fail a Test

You can make some mistakes while performing an A/B test./n/nLet's study them:

Heterogeneous audience

We have to perform the test with both groups at the same time. Imagine you want to test the buttons for the one group before the holiday and the second group during the holiday. Because of the holiday, there might be more customers, for example, searching for candies in the online candy shop.

To perform an excellent test, we must prepare two common groups for the test. Imagine you create English courses. If there are more English native speakers in the first group than in the second, conversions may be high in the first group, but it doesn't mean that the control variant of the test won!

Conversion is a process when a visitor user becomes a customer.

Peeking problem

The most challenging stage for the product has come - do not rush. A common mistake is to draw a conclusion before the time when the victory of the alternative hypothesis is visible (i.e., a positive change in the metric on option B). By the end of the experiment, the results may be reversed, so always bring the experiment to an end.

Ignoring minor results

During an A/B test, an increase in the monitored metric is detected but lower than expected. For example, we expected growth of 7-10%, and the metric grew by 2-3%. We have received valuable information that can lead us to a new hypothesis or problem.

Ignoring other indicators

It may well be that the experiment shows an increase in the metric we need, but other indicators are declining. It is necessary to conduct additional research and build hypotheses to solve the problem, but in no case give up.

Type I error

So in statistics is called the situation when there is an illusion of the result. For example, our hypothesis works, although there is no effect.

Type II error

The opposite situation is when we do not see fundamental changes. It often happens that the metric reacts to changes in the lack of data (i.e., few users for analysis).

How to cope with these problems? Not to perform these types of mistakes, you have to repeat testing, testing on all users, and qualitative research are carried out. Just as a doctor is never in a hurry to make a hasty diagnosis, so you should not jump to conclusions. Doubt is the key to finding working hypotheses.