Lära Chi-Square A/B | Statistical Analysis of A/B

When you run an A/B test with categorical outcomes—such as whether a user converted or not—the chi-square test provides a way to determine if the observed differences between groups are statistically significant. Unlike t-tests or z-tests, which are used for continuous or proportion data, the chi-square test is designed for count data in categories. In the context of A/B testing, you often want to know if the proportion of users who converted is different between the control and treatment groups, but without assuming anything about the underlying distribution of conversions.


              12345678910111213141516171819202122
            
import pandas as pd
from scipy.stats import chi2_contingency

# Example conversion data for A/B groups
data = {
    "Group": ["Control", "Control", "Treatment", "Treatment"],
    "Outcome": ["Converted", "Not Converted", "Converted", "Not Converted"],
    "Count": [120, 880, 150, 850]
}

df = pd.DataFrame(data)

# Create a contingency table
contingency_table = df.pivot(index="Group", columns="Outcome", values="Count")

# Run the chi-square test
chi2, p, dof, expected = chi2_contingency(contingency_table)

print(f"Chi-square statistic: {chi2:.2f}")
print(f"p-value: {p:.4f}")
print("Expected frequencies:")
print(pd.DataFrame(expected, index=contingency_table.index, columns=contingency_table.columns))

Interpreting Chi-Square Test Results

When you run the chi-square test, focus on these key outputs:

Chi-square statistic: measures how much the observed counts deviate from the expected counts if there were no difference between groups;
p-value: shows how likely it is to observe a difference as large as the one in your data, assuming no true difference exists.

To make a decision:

If the p-value is below your significance threshold (commonly 0.05), you can conclude there is a statistically significant difference in conversion rates between the control and treatment groups;
If the p-value is above the threshold, you do not have enough evidence to claim a difference.

Important:

A significant result tells you a difference exists, but not which group performed better or by how much;
Always check the observed and expected frequencies to confirm the test's assumptions are met;
Use the chi-square test alongside other analyses for a complete understanding of your A/B test outcomes.

Var allt tydligt?

Tack för dina kommentarer!

Avsnitt 5. Kapitel 3

Fråga AI

Fråga vad du vill eller prova någon av de föreslagna frågorna för att starta vårt samtal

Suggested prompts:

Can you explain how to interpret the expected frequencies in the output?

What should I do if my p-value is very close to 0.05?

How do I know if the chi-square test assumptions are met in my data?

Awesome!

Completion rate improved to 3.23

Svep för att visa menyn


              12345678910111213141516171819202122
            
import pandas as pd
from scipy.stats import chi2_contingency

# Example conversion data for A/B groups
data = {
    "Group": ["Control", "Control", "Treatment", "Treatment"],
    "Outcome": ["Converted", "Not Converted", "Converted", "Not Converted"],
    "Count": [120, 880, 150, 850]
}

df = pd.DataFrame(data)

# Create a contingency table
contingency_table = df.pivot(index="Group", columns="Outcome", values="Count")

# Run the chi-square test
chi2, p, dof, expected = chi2_contingency(contingency_table)

print(f"Chi-square statistic: {chi2:.2f}")
print(f"p-value: {p:.4f}")
print("Expected frequencies:")
print(pd.DataFrame(expected, index=contingency_table.index, columns=contingency_table.columns))