Apprendre t-test A/B | Statistical Analysis of A/B

In A/B testing, you want to determine whether there is a statistically significant difference between the means of a control group and a treatment group.

The t-test is a widely used statistical method for this purpose when:

The sample sizes are moderate;
The population standard deviation is unknown.

The core idea:

Compare the average outcome (such as conversion rate, revenue per user, or time on site) between the two groups;
If the difference in means is large enough relative to the variability in the data, you can conclude that the treatment had a real effect.


              12345678910111213141516171819202122
            
import pandas as pd
from scipy import stats

# Example experiment data
data = {
    "group": ["control"] * 100 + ["treatment"] * 100,
    "metric": (
        [1.2, 1.4, 1.5, 1.1, 1.3] * 20 +   # Control group metrics
        [1.6, 1.7, 1.8, 1.5, 1.9] * 20     # Treatment group metrics
    )
}
df = pd.DataFrame(data)

# Split into control and treatment
control = df[df["group"] == "control"]["metric"]
treatment = df[df["group"] == "treatment"]["metric"]

# Perform independent two-sample t-test
t_stat, p_value = stats.ttest_ind(control, treatment, equal_var=False)

print("t-statistic:", t_stat)
print("p-value:", p_value)

After running a t-test on your A/B experiment data, you receive two key outputs:

t-statistic: measures how large the difference between group means is, relative to the variability in your data;
p-value: shows the probability of seeing such a difference (or a more extreme one) if there were actually no real effect between the groups.

In A/B testing:

If the p-value is less than your chosen significance level (commonly 0.05), you can reject the null hypothesis;
This suggests that the treatment likely caused a real change in your metric.

Always interpret these results in the context of your business goals, the size of the effect, and the assumptions behind the t-test.

Tout était clair ?

Merci pour vos commentaires !

Section 5. Chapitre 1

Demandez à l'IA

Posez n'importe quelle question ou essayez l'une des questions suggérées pour commencer notre discussion

Awesome!

Completion rate improved to 3.23

Glissez pour afficher le menu