Learn Simulating A/B Test Data | Practical Analysis, Interpretation, and Reporting

Swipe to show menu

Simulating A/B test data is a valuable skill for anyone learning about experimentation and analysis. When you generate synthetic datasets, you can practice statistical techniques, test your analysis workflow, and experiment with different scenarios without needing access to real user data. Synthetic data is especially useful for learning because it allows you to control key parameters, such as group sizes and conversion rates, and to repeat experiments under known conditions. This makes it easier to understand the impact of various factors on your results and to develop your analytical skills in a risk-free environment.


              1234567891011121314151617181920212223242526272829303132333435363738394041
            
import numpy as np
import pandas as pd

# Set random seed for reproducibility
np.random.seed(42)

# Define number of users per group
n_users = 1000

# Define conversion rates for group A and B
conversion_rate_A = 0.10  # 10%
conversion_rate_B = 0.13  # 13%

# Generate user IDs
user_ids = np.arange(1, 2 * n_users + 1)

# Randomly assign users to groups
groups = np.array(['A'] * n_users + ['B'] * n_users)
np.random.shuffle(groups)

# Assign conversions based on group-specific rates
conversions = []
for group in groups:
    if group == 'A':
        conversions.append(np.random.binomial(1, conversion_rate_A))
    else:
        conversions.append(np.random.binomial(1, conversion_rate_B))

# Create DataFrame
data = pd.DataFrame({
    'user_id': user_ids,
    'group': groups,
    'converted': conversions
})

# Show the first few rows
print(data.head())

# To adjust for different scenarios:
# - Change n_users for sample size
# - Modify conversion_rate_A or conversion_rate_B for different effect sizes

After generating your simulated A/B test data, it is important to validate that the dataset matches your intended scenario. First, check that the number of users in each group is balanced, or as expected for your design. Next, calculate the observed conversion rates for each group to ensure they are close to your specified rates. You should also review the dataset for any missing or duplicate entries, and verify that every user has a valid group assignment and outcome. This validation step ensures your synthetic data is realistic and reliable for practicing analysis.

Everything was clear?

Thanks for your feedback!

Section 4. Chapter 1

Ask AI

Ask anything or try one of the suggested questions to begin our chat

Section 4. Chapter 1