Learn Visualizing A/B Test Outcomes | Practical Analysis, Interpretation, and Reporting

Swipe to show menu

When presenting A/B test results, your visualizations must make the key findings instantly clear and prevent misinterpretation. The most effective approaches include:

Bar plots: these quickly compare conversion rates or other metrics between groups;
Bar plots with error bars: adding confidence intervals to bars helps viewers understand the uncertainty in your estimates;
Distribution plots: displaying the full distribution of metrics (such as conversion rates or revenue per user) for each group can reveal differences in variability or outliers;
Line plots: useful when tracking changes over time or across multiple test periods.

A good visualization uses clearly labeled axes, consistent color schemes, and highlights the most important differences. For example, a bar plot showing conversion rates for group A and B, with error bars for 95% confidence intervals, allows the audience to see both the central estimate and the uncertainty.

In contrast, a poor visualization might use misleading y-axis scales (such as truncating the axis to exaggerate differences), fail to label axes or groups, or use distracting colors and cluttered legends. These mistakes can confuse the audience or even mislead them about the test's outcome.


              123456789101112131415161718192021222324252627282930
            
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np

# Example conversion rates and confidence intervals
groups = ['A', 'B']
conversion_rates = [0.12, 0.15]
conf_intervals = [(0.10, 0.14), (0.13, 0.17)]  # 95% confidence intervals

# Calculate error bars (distance from mean to CI bounds)
errors = [
    [conversion_rates[0] - conf_intervals[0][0], conf_intervals[0][1] - conversion_rates[0]],
    [conversion_rates[1] - conf_intervals[1][0], conf_intervals[1][1] - conversion_rates[1]]
]
errors = np.array(errors).T  # shape (2, 2) for matplotlib

fig, ax = plt.subplots(figsize=(6, 4))
bars = ax.bar(groups, conversion_rates, yerr=errors, capsize=10, color=['#4C72B0', '#55A868'])

ax.set_ylabel('Conversion Rate')
ax.set_title('A/B Test Conversion Rates with 95% Confidence Intervals')
ax.set_ylim(0, 0.2)
ax.yaxis.set_major_formatter(plt.FuncFormatter(lambda y, _: '{:.0%}'.format(y)))

# Add value labels on top of bars
for bar, rate in zip(bars, conversion_rates):
    ax.text(bar.get_x() + bar.get_width() / 2, rate + 0.005, f"{rate:.2%}", ha='center', va='bottom', fontsize=11)

plt.tight_layout()
plt.show()

When choosing a visualization, consider your audience and the story you want to tell:

For executives or non-technical stakeholders: use clear bar plots with confidence intervals, minimal clutter, and direct labels to highlight the main takeaway;
For analysts or data scientists: supplement bar plots with distribution plots (such as violin or box plots) to show the full range of outcomes and variability;
For presentations or reports: avoid misleading elements like truncated axes, ambiguous group labels, or unnecessary 3D effects. Always provide context and explain what the error bars or distributions represent.

Matching the visualization style to your audience ensures your results are understood and trusted.

Everything was clear?

Thanks for your feedback!

Section 4. Chapter 3

Ask AI

Ask anything or try one of the suggested questions to begin our chat

Section 4. Chapter 3