Learn Metrics and Success Criteria | Designing Effective A/B Tests

Swipe to show menu

When designing an A/B test, choosing the right metrics is crucial to determining whether your experiment is successful. Metrics are measurable values that reflect user behavior or business outcomes. Some of the most common A/B test metrics include:

Definition

Conversion Rate - the percentage of users who complete a desired action, such as making a purchase or signing up for a newsletter.

Definition

Click-Through Rate (CTR) - the proportion of users who click on a specific link or button out of those who view it.

Definition

Revenue per User - the average amount of revenue generated per user during the test period.

Selecting which metric to use depends on your business objectives. For example, if your goal is to increase sales, conversion rate or revenue per user are strong choices. If you want to boost engagement, click-through rate or time on site might be more relevant.

Good metric choices are those that are closely tied to your business goals and are sensitive enough to detect meaningful changes. If you run an e-commerce site and launch a new checkout flow, measuring the conversion rate from cart to purchase is a direct indicator of success.

Bad metric choices occur when you select metrics that are not aligned with your objectives, are too broad, or are easily manipulated. Measuring page views when your goal is to increase purchases can be misleading - users may view more pages without actually buying anything.


              12345678910111213141516
            
import pandas as pd

# Sample data: user actions from an A/B test
data = {
    "user_id": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    "group": ["A", "A", "B", "B", "A", "B", "A", "B", "A", "B"],
    "converted": [0, 1, 1, 0, 0, 1, 1, 0, 1, 1]  # 1 = purchase, 0 = no purchase
}
df = pd.DataFrame(data)

# Calculating conversion rate for each group
conversion_rates = df.groupby("group")["converted"].mean()

# Printing results with business context
print("Conversion Rate for Group A:", round(conversion_rates["A"] * 100, 2), "%")
print("Conversion Rate for Group B:", round(conversion_rates["B"] * 100, 2), "%")

Defining success criteria means setting clear thresholds that determine whether the test outcome is meaningful for your business. Instead of just asking "Did the metric go up?", specify by how much it should increase to be considered a win. You might decide that a new feature is successful only if it increases conversion rate by at least 2%.

Note

You should also consider the broader business impact. Sometimes a small improvement in your primary metric can have a large effect on revenue or user satisfaction, while in other cases, the change might not be worth the implementation cost.

Be cautious about using vanity metrics - numbers that look good on paper but don't reflect real business value. An increase in app downloads is only valuable if those users actually engage with your product or make purchases.

Primary metrics are the main indicators of success and should be directly tied to your hypothesis;
Secondary metrics can provide supporting evidence or help catch unintended side effects, but they should not distract from your main goal.

Always ensure your metrics are actionable, aligned with your objectives, and resistant to manipulation.

Everything was clear?

Thanks for your feedback!

Section 2. Chapter 3

Ask AI

Ask anything or try one of the suggested questions to begin our chat

Section 2. Chapter 3