Learn Control vs. Variant | Experimentation and A/B Testing

Swipe to show menu

A/B testing is a fundamental technique in product analytics, allowing you to compare the impact of a new feature or change against the current experience. In an A/B test, users are split into two groups: the control group and the variant group. The control group experiences the product as usual, while the variant group receives the new feature or change you want to test.

Imagine you are testing a new checkout button color in an e-commerce app. The control group sees the original button color, while the variant group sees the new color. By measuring outcomes - such as completed purchases - you can determine whether the new button color has a positive, negative, or no effect on user behavior.

Note

Random assignment to control and variant groups helps ensure unbiased results. This means that any observed differences are more likely due to the change being tested, not pre-existing differences between users.


              12345678910111213141516171819202122232425262728293031323334
            
import numpy as np
import pandas as pd

# Simulate 2000 users
np.random.seed(42)
user_ids = np.arange(1, 2001)

# Randomly assign users to control or variant
groups = np.random.choice(["control", "variant"], size=2000)

# Simulate conversion: higher for variant group
conversion_prob = np.where(groups == "control", 0.12, 0.15)
converted = np.random.binomial(1, conversion_prob)

# Simulate purchase value for those who converted
purchase_value = np.where(converted == 1,
                         np.random.normal(loc=np.where(groups == "control", 45, 47), scale=5),
                         0)

# Create DataFrame
ab_test_results = pd.DataFrame({
    "user_id": user_ids,
    "group": groups,
    "converted": converted,
    "purchase_value": purchase_value
})

# Show summary statistics
summary = ab_test_results.groupby("group").agg(
    users=("user_id", "count"),
    conversion_rate=("converted", "mean"),
    avg_purchase_value=("purchase_value", lambda x: x[x > 0].mean())
)
print(summary)

After running your A/B test and collecting data, you will compare the results between the control and variant groups. Key metrics to examine include conversion rate and average purchase value. You are looking for meaningful differences that suggest the new feature is having a real impact. If the variant group shows higher conversion or revenue, and the assignment was random, you can be more confident that the change is responsible for the improvement.

1. Why is random assignment important in A/B testing?

2. Fill in the blank:

Everything was clear?

Thanks for your feedback!

Section 4. Chapter 2

Ask AI

Ask anything or try one of the suggested questions to begin our chat

Section 4. Chapter 2