Notice: This page requires JavaScript to function properly.
Please enable JavaScript in your browser settings or update your browser.Learn Intervals to Compare | The First A/B Test
A/B Testing in Python

book
Intervals to Compare

In the previous chapter, we created 2 plots. We can also create 2 confidence intervals for these groups.

A confidence interval is the mean of your estimate plus and minus the variation in that estimate. This is the range of values you expect your estimate to fall between if you redo your test, within a certain level of confidence. Confidence, in statistics, is another way to describe probability.

Tu build them use scipy.stats.t.interval(alpha, data, loc, scale). In our case we will use alpha equals 0.95(you may also choose 0,99, but you will need to compare the p-value with 0,01 thus), the data.shape[1] as a data, loc = data.clicks.mean() and scale = scipy.stats.sem(data.clicks).

If intervals cover each other a lot, 2 groups don't differ a lot => the new version of the site doesn't make any big changes.

Task

Swipe to start coding

  1. Build the confidence interval for the df_control using the information from the NOTE in the theory.
  2. Build the confidence interval for the df_test.

Solution

# Importing the pandas
import pandas as pd
# Importing the seaborn
import seaborn as sns
# Importing the scipy
import scipy

# Reading the file
df = pd.read_csv('https://codefinity-content-media.s3.eu-west-1.amazonaws.com/ae14b913-9d96-48cb-ace7-a332315f7cf4/ab+test+1.csv')
# Control group
df_control = df.query('group == "control" ')
# Test group
df_test = df.query('group == "test" ')

# Build the interval for the df_control
print(scipy.stats.t.interval(0.95, df_control.shape[1], loc = df_control.clicks.mean(), scale = scipy.stats.sem(df_control.clicks)))
# Build the interval for the df_test
print(scipy.stats.t.interval(0.95, df_test.shape[1], loc = df_test.clicks.mean(), scale = scipy.stats.sem(df_test.clicks)))

Everything was clear?

How can we improve it?

Thanks for your feedback!

Section 2. Chapter 6
# Importing the pandas
import pandas as pd
# Importing the seaborn
import seaborn as sns
# Importing the scipy
import scipy

# Reading the file
df = pd.read_csv('https://codefinity-content-media.s3.eu-west-1.amazonaws.com/ae14b913-9d96-48cb-ace7-a332315f7cf4/ab+test+1.csv')
# Control group
df_control = df.query('group == "control" ')
# Test group
df_test = df.query('group == "test" ')

# Build the interval for the df_control
print(___(0.95, df_control.shape[1], loc = ___.clicks.mean(), scale = scipy.stats.sem(___.clicks)))
# Build the interval for the df_test
___
toggle bottom row
some-alt