Lære Confidence Intervals | Statistical Analysis of A/B

Understanding confidence intervals is essential for interpreting the results of A/B tests. A confidence interval provides a range of values that is likely to contain the true value of a population parameter, such as a mean or proportion, with a certain level of confidence (commonly 95%). Instead of reporting only a single estimate, like the difference in conversion rates between two groups, you use a confidence interval to express the uncertainty around that estimate. This helps you understand not just the observed effect, but also the possible range of the true effect based on your data.

A confidence interval for the mean is typically calculated as:

\bar{x} \pm t^* \cdot \frac{s}{\sqrt{n}}

Where:

$\bar{x}$ is the sample mean;
$t^*$ is the critical value from the t-distribution;
$s$ is the sample standard deviation;
$n$ is the sample size.

For a proportion, the confidence interval is:

p \pm z^* \cdot \sqrt{\frac{p(1-p)}{n}}

Where:

$p$ is the sample proportion;
$z^*$ is the critical value from the normal distribution;
$n$ is the sample size.

In A/B testing, confidence intervals are especially important because they help you determine whether a result is statistically significant and how much you can trust the observed difference. If the confidence interval for the difference between groups does not include zero, you have evidence that the groups are truly different at your chosen confidence level. However, if zero is within the interval, the difference may be due to random chance.


              123456789101112131415161718192021
            
import numpy as np
from scipy import stats

# Confidence interval for the mean
data = [12, 14, 15, 16, 14, 13, 15, 16, 17, 18]
confidence = 0.95
mean = np.mean(data)
sem = stats.sem(data)
interval = stats.t.interval(confidence, len(data)-1, loc=mean, scale=sem)
print("Confidence interval for the mean:", interval)

# Confidence interval for a proportion
# Suppose 200 users, 54 converted (successes)
n = 200
successes = 54
p_hat = successes / n
z = stats.norm.ppf(1 - (1-confidence)/2)
se = np.sqrt(p_hat * (1 - p_hat) / n)
lower = p_hat - z * se
upper = p_hat + z * se
print("Confidence interval for the proportion:", (lower, upper))

Alt var klart?

Takk for tilbakemeldingene dine!

Seksjon 5. Kapittel 4

Spør AI

Spør om hva du vil, eller prøv ett av de foreslåtte spørsmålene for å starte chatten vår

Awesome!

Completion rate improved to 3.23

Sveip for å vise menyen

A confidence interval for the mean is typically calculated as:

\bar{x} \pm t^* \cdot \frac{s}{\sqrt{n}}

Where:

$\bar{x}$ is the sample mean;
$t^*$ is the critical value from the t-distribution;
$s$ is the sample standard deviation;
$n$ is the sample size.

For a proportion, the confidence interval is:

p \pm z^* \cdot \sqrt{\frac{p(1-p)}{n}}

Where:

$p$ is the sample proportion;
$z^*$ is the critical value from the normal distribution;
$n$ is the sample size.


              123456789101112131415161718192021
            
import numpy as np
from scipy import stats

# Confidence interval for the mean
data = [12, 14, 15, 16, 14, 13, 15, 16, 17, 18]
confidence = 0.95
mean = np.mean(data)
sem = stats.sem(data)
interval = stats.t.interval(confidence, len(data)-1, loc=mean, scale=sem)
print("Confidence interval for the mean:", interval)

# Confidence interval for a proportion
# Suppose 200 users, 54 converted (successes)
n = 200
successes = 54
p_hat = successes / n
z = stats.norm.ppf(1 - (1-confidence)/2)
se = np.sqrt(p_hat * (1 - p_hat) / n)
lower = p_hat - z * se
upper = p_hat + z * se
print("Confidence interval for the proportion:", (lower, upper))

Alt var klart?

Takk for tilbakemeldingene dine!

Seksjon 5. Kapittel 4