Apprendre z-test Conversion | Statistical Analysis of A/B

When you want to compare the conversion rates between two groups in an A/B test, the z-test for proportions is a powerful and commonly used tool. This test helps you determine if the difference in conversion rates between your control and treatment groups is statistically significant, or if it could have happened by random chance. You use this test when your outcome variable is binary (such as converted or did not convert) and you have large enough sample sizes in each group.

The z-test for conversion rates works by comparing the observed proportions of conversions in each group. The null hypothesis typically states that there is no difference between the two conversion rates, while the alternative hypothesis claims that there is a difference. The test statistic is calculated using the observed proportions and sample sizes, and then compared to a standard normal distribution to obtain a p-value.

To run a z-test for conversion rates, you need to know:

The number of users in each group;
The number of conversions in each group;
The observed conversion rates (conversions divided by group size).


              123456789101112131415161718192021222324252627282930313233343536
            
import numpy as np
from scipy.stats import norm

# Control group data
n_control = 1000  # total users in control group
x_control = 120   # conversions in control group

# Treatment group data
n_treatment = 980  # total users in treatment group
x_treatment = 138  # conversions in treatment group

# Calculate observed conversion rates
p_control = x_control / n_control
p_treatment = x_treatment / n_treatment

# Pooled conversion rate
p_pooled = (x_control + x_treatment) / (n_control + n_treatment)

# Standard error
se = np.sqrt(p_pooled * (1 - p_pooled) * (1/n_control + 1/n_treatment))

# z-score
z = (p_treatment - p_control) / se

# Two-tailed p-value
p_value = 2 * (1 - norm.cdf(abs(z)))

print(f"Control conversion rate: {p_control:.3f}")
print(f"Treatment conversion rate: {p_treatment:.3f}")
print(f"z-score: {z:.3f}")
print(f"p-value: {p_value:.4f}")

if p_value < 0.05:
    print("Statistically significant difference in conversion rates.")
else:
    print("No statistically significant difference detected.")

Tout était clair ?

Merci pour vos commentaires !

Section 5. Chapitre 2

Demandez à l'IA

Posez n'importe quelle question ou essayez l'une des questions suggérées pour commencer notre discussion

Suggested prompts:

Can you explain what the z-score and p-value mean in this context?

What are the assumptions behind using a z-test for proportions?

How do I interpret the result if the p-value is above 0.05?

Awesome!

Completion rate improved to 3.23

Glissez pour afficher le menu

To run a z-test for conversion rates, you need to know:

The number of users in each group;
The number of conversions in each group;
The observed conversion rates (conversions divided by group size).


              123456789101112131415161718192021222324252627282930313233343536
            
import numpy as np
from scipy.stats import norm

# Control group data
n_control = 1000  # total users in control group
x_control = 120   # conversions in control group

# Treatment group data
n_treatment = 980  # total users in treatment group
x_treatment = 138  # conversions in treatment group

# Calculate observed conversion rates
p_control = x_control / n_control
p_treatment = x_treatment / n_treatment

# Pooled conversion rate
p_pooled = (x_control + x_treatment) / (n_control + n_treatment)

# Standard error
se = np.sqrt(p_pooled * (1 - p_pooled) * (1/n_control + 1/n_treatment))

# z-score
z = (p_treatment - p_control) / se

# Two-tailed p-value
p_value = 2 * (1 - norm.cdf(abs(z)))

print(f"Control conversion rate: {p_control:.3f}")
print(f"Treatment conversion rate: {p_treatment:.3f}")
print(f"z-score: {z:.3f}")
print(f"p-value: {p_value:.4f}")

if p_value < 0.05:
    print("Statistically significant difference in conversion rates.")
else:
    print("No statistically significant difference detected.")

Tout était clair ?

Merci pour vos commentaires !

Section 5. Chapitre 2