Summary  
Calculating confidence intervals by determining sample estimates (mean or proportion), computing their standard errors, obtaining critical values from the t- or normal distribution, and then deriving the lower and upper bounds.

General domain of usage  
A/B testing

Understanding **confidence intervals** is essential for interpreting the results of A/B tests. A **confidence interval** provides a range of values that is likely to contain the true value of a population parameter, such as a mean or proportion, with a certain level of confidence (commonly 95%). Instead of reporting only a single estimate, like the difference in conversion rates between two groups, you use a **confidence interval** to express the uncertainty around that estimate. This helps you understand not just the observed effect, but also the possible range of the true effect based on your data.

A confidence interval for the mean is typically calculated as:

$$
\bar{x} \pm t^* \cdot \frac{s}{\sqrt{n}}
$$

Where:
- $$\bar{x}$$ is the sample mean;
- $$t^*$$ is the critical value from the t-distribution;
- $$s$$ is the sample standard deviation;
- $$n$$ is the sample size.

For a proportion, the confidence interval is:

$$
p \pm z^* \cdot \sqrt{\frac{p(1-p)}{n}}
$$

Where:
- $$p$$ is the sample proportion;
- $$z^*$$ is the critical value from the normal distribution;
- $$n$$ is the sample size.

In A/B testing, **confidence intervals** are especially important because they help you determine whether a result is statistically significant and how much you can trust the observed difference. If the **confidence interval** for the difference between groups does not include zero, you have evidence that the groups are truly different at your chosen confidence level. However, if zero is within the interval, the difference may be due to random chance.

import numpy as np
from scipy import stats

# Confidence interval for the mean
data = [12, 14, 15, 16, 14, 13, 15, 16, 17, 18]
confidence = 0.95
mean = np.mean(data)
sem = stats.sem(data)
interval = stats.t.interval(confidence, len(data)-1, loc=mean, scale=sem)
print("Confidence interval for the mean:", interval)

# Confidence interval for a proportion
# Suppose 200 users, 54 converted (successes)
n = 200
successes = 54
p_hat = successes / n
z = stats.norm.ppf(1 - (1-confidence)/2)
se = np.sqrt(p_hat * (1 - p_hat) / n)
lower = p_hat - z * se
upper = p_hat + z * se
print("Confidence interval for the proportion:", (lower, upper))

How do confidence intervals help you make decisions in A/B testing experiments?

Master the principles and practical application of hypothesis testing and A/B testing. Learn statistical tests, experiment design, data preparation, and analysis techniques using Python to make data-driven decisions in real-world scenarios.

Explore the core concepts of hypothesis testing, including t-tests, hypothesis formulation, and statistical assumptions, with practical Python examples.

Learn about z-tests for means and proportions, chi-square tests, and how to select the appropriate statistical test for different data types.

Build a strong foundation in A/B testing by understanding experiment structure, metric selection, randomization, and sample size considerations.

Learn how to structure, clean, and validate experimental data, including outlier detection and metric construction for robust analysis.

Apply statistical tests to A/B test data, interpret results, and understand key concepts like confidence intervals, uplift, and p-values.

Understand and implement guardrail metrics, monitor variance, and apply stopping rules to ensure experiment integrity and reliability.

Confidence Intervals