Lära t-test A/B

Svep för att visa menyn

In A/B testing, you want to determine whether there is a statistically significant difference between the means of a control group and a treatment group.

The t-test is a widely used statistical method for this purpose when:

The sample sizes are moderate;
The population standard deviation is unknown.

The core idea:

Compare the average outcome (such as conversion rate, revenue per user, or time on site) between the two groups;
If the difference in means is large enough relative to the variability in the data, you can conclude that the treatment had a real effect.


              12345678910111213141516171819202122
            
import pandas as pd
from scipy import stats

# Example experiment data
data = {
    "group": ["control"] * 100 + ["treatment"] * 100,
    "metric": (
        [1.2, 1.4, 1.5, 1.1, 1.3] * 20 +   # Control group metrics
        [1.6, 1.7, 1.8, 1.5, 1.9] * 20     # Treatment group metrics
    )
}
df = pd.DataFrame(data)

# Split into control and treatment
control = df[df["group"] == "control"]["metric"]
treatment = df[df["group"] == "treatment"]["metric"]

# Perform independent two-sample t-test
t_stat, p_value = stats.ttest_ind(control, treatment, equal_var=False)

print("t-statistic:", t_stat)
print("p-value:", p_value)

After running a t-test on your A/B experiment data, you receive two key outputs:

t-statistic: measures how large the difference between group means is, relative to the variability in your data;
p-value: shows the probability of seeing such a difference (or a more extreme one) if there were actually no real effect between the groups.

In A/B testing:

If the p-value is less than your chosen significance level (commonly 0.05), you can reject the null hypothesis;
This suggests that the treatment likely caused a real change in your metric.

Always interpret these results in the context of your business goals, the size of the effect, and the assumptions behind the t-test.

Var allt tydligt?

Tack för dina kommentarer!

Avsnitt 1. Kapitel 23

Fråga AI

Fråga vad du vill eller prova någon av de föreslagna frågorna för att starta vårt samtal

Avsnitt 1. Kapitel 23