So, we have pairwise compared both datasets' columns. Let's recall Section 1. We need a **metric**, or better yet, multiple metrics. Good metrics for our datasets would be:




Let's compare the first metric, **Conversion Rate**, for both datasets. We will plot **histograms**:

# Import libraries
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns

# Read .csv files 
df_control = pd.read_csv('https://codefinity-content-media.s3.eu-west-1.amazonaws.com/c3b98ad3-420d-403f-908d-6ab8facc3e28/ab_control.csv', delimiter=';')
df_test = pd.read_csv('https://codefinity-content-media.s3.eu-west-1.amazonaws.com/c3b98ad3-420d-403f-908d-6ab8facc3e28/ab_test.csv', delimiter=';')

# Define metric
df_test['Conversion'] = df_test['Purchase'] / df_test['Click']
df_control['Conversion'] = df_control['Purchase'] / df_control['Click']

# Ploting hists
sns.histplot(df_control['Conversion'], color="#1e2635", label="Conversion of Control Group")
sns.histplot(df_test['Conversion'], color="#ff8a00", label="Conversion of Test Group")

# Add mean line
plt.axvline(df_control['Conversion'].mean(), color="#1e2635", linestyle='dashed', linewidth=1, label='Mean Control Group')
plt.axvline(df_test['Conversion'].mean(), color="#ff8a00", linestyle='dashed', linewidth=1, label='Mean Test Group')

# Sign the axes
plt.xlabel('Conversion')
plt.ylabel('Frequency')
plt.legend()
plt.title('Histogram of Conversion')

# Show the results
plt.show()

Well, it **doesn't** seem to follow a **normal** distribution. Let's plot a **box plot**:

#Import libraries
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns

#Read .csv files 
df_control = pd.read_csv('https://codefinity-content-media.s3.eu-west-1.amazonaws.com/c3b98ad3-420d-403f-908d-6ab8facc3e28/ab_control.csv', delimiter=';')
df_test = pd.read_csv('https://codefinity-content-media.s3.eu-west-1.amazonaws.com/c3b98ad3-420d-403f-908d-6ab8facc3e28/ab_test.csv', delimiter=';')

#Define metric
df_test['Conversion'] = df_test['Purchase'] / df_test['Click']
df_control['Conversion'] = df_control['Purchase'] / df_control['Click']

#We add to the dataframes columns-labels, which mean belonging to either the control or the test group
df_control['group'] = 'Contol group'
df_test['group'] = 'Test group'

#Concat the dataframes and plotting boxplots
df_combined = pd.concat([df_control, df_test])
sns.boxplot(data=df_combined, x='group', y='Conversion', palette=['#1e2635', '#ff8a00'],
            medianprops={'color': 'red'})

#Sign the axis 
plt.xlabel('')
plt.ylabel('Conversion')
plt.title('Comparison of Conversion')

#Show the results
plt.show()

The distributions are heavily skewed, suggesting they are **unlikely** to be **normal**. Let's confirm this by performing the **Shapiro-Wilk test**:

# Import libraries
import pandas as pd
from scipy.stats import shapiro

# Read .csv files 
df_control = pd.read_csv('https://codefinity-content-media.s3.eu-west-1.amazonaws.com/c3b98ad3-420d-403f-908d-6ab8facc3e28/ab_control.csv', delimiter=';')
df_test = pd.read_csv('https://codefinity-content-media.s3.eu-west-1.amazonaws.com/c3b98ad3-420d-403f-908d-6ab8facc3e28/ab_test.csv', delimiter=';')

# Define metric
df_test['Conversion'] = df_test['Purchase'] / df_test['Click']
df_control['Conversion'] = df_control['Purchase'] / df_control['Click']

# Testing normality
stat_control, p_control = shapiro(df_control['Conversion'])
print("Control group: ")
print("Stat: %.4f, p-value: %.4f" % (stat_control, p_control))
if p_control > 0.05:
  print('Control group is likely to normal distribution')
else:
  print('Control group is NOT likely to normal distribution')

stat_control, p_control = shapiro(df_test['Conversion'])
print("Test group: ")
print("Stat: %.4f, p-value: %.4f" % (stat_control, p_control))
if p_control > 0.05:
  print('Control group is likely to normal distribution')
else:
  print('Control group is NOT likely to normal distribution')

The **Shapiro-Wilk test** did not provide sufficient statistical evidence for the normality of the Conversion metric distributions. However, this does not hinder us. Even in such a situation, we can turn to the non-parametric **Mann-Whitney U-test**, also known as the **U-test**.

This course is a comprehensive and practical online program designed to equip individuals with the knowledge and skills necessary to conduct effective A/B tests. 

In this course, participants will learn the fundamental principles of A/B testing, including experimental design, sample size determination, hypothesis testing, and statistical analysis.

Welcome to the A/B testing course. In this section, we will tell you what controlled experiments are, where they are used, and why they are used. We will look at a specific example of a controlled experiment and talk about A/A testing. The Python programming language will help us with this. Well, let's get started!

Before conducting a controlled experiment, we need to determine the characteristics of the sample. The type of tests that we will conduct a little later will depend on these characteristics. You will learn to build distribution histograms, box plots, and check the normality of the distribution. Let's get started right now!

The last thing to do before conducting a controlled experiment is to check the sampling variances. Some types of graphs and Levene's test will help us with this. Let's talk about it in more detail in this section!

So we got to A/B testing. Soon you will make your first parametric tests, or as they are also called - T-tests. We'll use all the knowledge we've learned in the previous sections: metric definition, normality testing, variance testing, and plotting. All this will help us get statistically significant results. Well, to work!

It turns out that sometimes the samples do not have a normal distribution. This means that we cannot perform parametric testing. What to do in such a situation? Non-parametric testing. We will talk about him in the last section.

The Art of A/B Testing

Metrics