Impara Confidence Intervals | Statistical Analysis of A/B

Understanding confidence intervals is essential for interpreting the results of A/B tests. A confidence interval provides a range of values that is likely to contain the true value of a population parameter, such as a mean or proportion, with a certain level of confidence (commonly 95%). Instead of reporting only a single estimate, like the difference in conversion rates between two groups, you use a confidence interval to express the uncertainty around that estimate. This helps you understand not just the observed effect, but also the possible range of the true effect based on your data.

A confidence interval for the mean is typically calculated as:

\bar{x} \pm t^* \cdot \frac{s}{\sqrt{n}}

Where:

$\bar{x}$ is the sample mean;
$t^*$ is the critical value from the t-distribution;
$s$ is the sample standard deviation;
$n$ is the sample size.

For a proportion, the confidence interval is:

p \pm z^* \cdot \sqrt{\frac{p(1-p)}{n}}

Where:

$p$ is the sample proportion;
$z^*$ is the critical value from the normal distribution;
$n$ is the sample size.

In A/B testing, confidence intervals are especially important because they help you determine whether a result is statistically significant and how much you can trust the observed difference. If the confidence interval for the difference between groups does not include zero, you have evidence that the groups are truly different at your chosen confidence level. However, if zero is within the interval, the difference may be due to random chance.


              123456789101112131415161718192021
            
import numpy as np
from scipy import stats

# Confidence interval for the mean
data = [12, 14, 15, 16, 14, 13, 15, 16, 17, 18]
confidence = 0.95
mean = np.mean(data)
sem = stats.sem(data)
interval = stats.t.interval(confidence, len(data)-1, loc=mean, scale=sem)
print("Confidence interval for the mean:", interval)

# Confidence interval for a proportion
# Suppose 200 users, 54 converted (successes)
n = 200
successes = 54
p_hat = successes / n
z = stats.norm.ppf(1 - (1-confidence)/2)
se = np.sqrt(p_hat * (1 - p_hat) / n)
lower = p_hat - z * se
upper = p_hat + z * se
print("Confidence interval for the proportion:", (lower, upper))

Tutto è chiaro?

Grazie per i tuoi commenti!

Sezione 5. Capitolo 4

Chieda ad AI

Chieda pure quello che desidera o provi una delle domande suggerite per iniziare la nostra conversazione

Awesome!

Completion rate improved to 3.23

Scorri per mostrare il menu

A confidence interval for the mean is typically calculated as:

\bar{x} \pm t^* \cdot \frac{s}{\sqrt{n}}

Where:

$\bar{x}$ is the sample mean;
$t^*$ is the critical value from the t-distribution;
$s$ is the sample standard deviation;
$n$ is the sample size.

For a proportion, the confidence interval is:

p \pm z^* \cdot \sqrt{\frac{p(1-p)}{n}}

Where:

$p$ is the sample proportion;
$z^*$ is the critical value from the normal distribution;
$n$ is the sample size.


              123456789101112131415161718192021
            
import numpy as np
from scipy import stats

# Confidence interval for the mean
data = [12, 14, 15, 16, 14, 13, 15, 16, 17, 18]
confidence = 0.95
mean = np.mean(data)
sem = stats.sem(data)
interval = stats.t.interval(confidence, len(data)-1, loc=mean, scale=sem)
print("Confidence interval for the mean:", interval)

# Confidence interval for a proportion
# Suppose 200 users, 54 converted (successes)
n = 200
successes = 54
p_hat = successes / n
z = stats.norm.ppf(1 - (1-confidence)/2)
se = np.sqrt(p_hat * (1 - p_hat) / n)
lower = p_hat - z * se
upper = p_hat + z * se
print("Confidence interval for the proportion:", (lower, upper))

Tutto è chiaro?

Grazie per i tuoi commenti!

Sezione 5. Capitolo 4