Learn Confidence Intervals | Statistical Foundations for A/B Testing

Swipe to show menu

Confidence intervals are a fundamental concept in statistics and play a crucial role in A/B testing. While p-values tell you whether an observed difference could be due to random chance, confidence intervals give you a range of values that likely contain the true effect size. This range helps you understand not just if there is a statistically significant difference, but also how big that difference might be and how certain you can be about it.

A confidence interval is calculated from your sample data and is typically expressed with a percentage, such as 95%. This means that if you repeated your experiment many times, 95% of the calculated intervals would contain the true population parameter. In A/B testing, you often use confidence intervals to estimate the difference in conversion rates between your control and variant groups.

The calculation of a confidence interval for a proportion (such as a conversion rate) involves determining the standard error of the observed rate, then using a z-score to define the range around the observed rate. Confidence intervals are more informative than p-values alone because they show both the magnitude and the precision of the estimated effect, allowing you to make better decisions about your test results.


              123456789101112131415161718192021222324252627282930313233343536
            
import numpy as np
from scipy.stats import norm

# Sample data: number of conversions and total users in each group
conversions_A = 200
users_A = 2000
conversions_B = 240
users_B = 2000

# Calculating conversion rates
rate_A = conversions_A / users_A
rate_B = conversions_B / users_B

# Calculatig the standard error for each group
se_A = np.sqrt(rate_A * (1 - rate_A) / users_A)
se_B = np.sqrt(rate_B * (1 - rate_B) / users_B)

# 95% confidence interval uses a z-score of approximately 1.96
z = norm.ppf(0.975)

# Calculating confidence intervals
ci_A = (rate_A - z * se_A, rate_A + z * se_A)
ci_B = (rate_B - z * se_B, rate_B + z * se_B)

print(f"Group A conversion rate: {rate_A:.3f}")
print(f"95% CI for Group A: ({ci_A[0]:.3f}, {ci_A[1]:.3f})")
print(f"Group B conversion rate: {rate_B:.3f}")
print(f"95% CI for Group B: ({ci_B[0]:.3f}, {ci_B[1]:.3f})")

# Confidence interval for the difference in conversion rates
diff = rate_B - rate_A
se_diff = np.sqrt(se_A**2 + se_B**2)
ci_diff = (diff - z * se_diff, diff + z * se_diff)

print(f"Difference in conversion rates (B - A): {diff:.3f}")
print(f"95% CI for difference: ({ci_diff[0]:.3f}, {ci_diff[1]:.3f})")

When you interpret a confidence interval in A/B testing, you are looking at the range where the true difference in conversion rates is likely to be. If the confidence interval for the difference does not include zero, you can be reasonably confident that there is a real effect. If it does include zero, the observed difference might be due to random chance.

For decision-making, confidence intervals help you understand both the possible size of the effect and the uncertainty around it. This makes it easier to communicate results to stakeholders: instead of saying only that a result is statistically significant, you can explain the likely range of improvement (or decline) and how confident you are in that estimate. This helps guide business decisions with a clearer understanding of potential risks and rewards.

Everything was clear?

Thanks for your feedback!

Section 3. Chapter 3

Ask AI

Ask anything or try one of the suggested questions to begin our chat

Section 3. Chapter 3