Leer Chi-Square A/B | Statistical Analysis of A/B

When you run an A/B test with categorical outcomes—such as whether a user converted or not—the chi-square test provides a way to determine if the observed differences between groups are statistically significant. Unlike t-tests or z-tests, which are used for continuous or proportion data, the chi-square test is designed for count data in categories. In the context of A/B testing, you often want to know if the proportion of users who converted is different between the control and treatment groups, but without assuming anything about the underlying distribution of conversions.


              12345678910111213141516171819202122
            
import pandas as pd
from scipy.stats import chi2_contingency

# Example conversion data for A/B groups
data = {
    "Group": ["Control", "Control", "Treatment", "Treatment"],
    "Outcome": ["Converted", "Not Converted", "Converted", "Not Converted"],
    "Count": [120, 880, 150, 850]
}

df = pd.DataFrame(data)

# Create a contingency table
contingency_table = df.pivot(index="Group", columns="Outcome", values="Count")

# Run the chi-square test
chi2, p, dof, expected = chi2_contingency(contingency_table)

print(f"Chi-square statistic: {chi2:.2f}")
print(f"p-value: {p:.4f}")
print("Expected frequencies:")
print(pd.DataFrame(expected, index=contingency_table.index, columns=contingency_table.columns))

Interpreting Chi-Square Test Results

When you run the chi-square test, focus on these key outputs:

Chi-square statistic: measures how much the observed counts deviate from the expected counts if there were no difference between groups;
p-value: shows how likely it is to observe a difference as large as the one in your data, assuming no true difference exists.

To make a decision:

If the p-value is below your significance threshold (commonly 0.05), you can conclude there is a statistically significant difference in conversion rates between the control and treatment groups;
If the p-value is above the threshold, you do not have enough evidence to claim a difference.

Important:

A significant result tells you a difference exists, but not which group performed better or by how much;
Always check the observed and expected frequencies to confirm the test's assumptions are met;
Use the chi-square test alongside other analyses for a complete understanding of your A/B test outcomes.

Was alles duidelijk?

Bedankt voor je feedback!

Sectie 5. Hoofdstuk 3

Vraag AI

Vraag wat u wilt of probeer een van de voorgestelde vragen om onze chat te starten.

Awesome!

Completion rate improved to 3.23

Veeg om het menu te tonen


              12345678910111213141516171819202122
            
import pandas as pd
from scipy.stats import chi2_contingency

# Example conversion data for A/B groups
data = {
    "Group": ["Control", "Control", "Treatment", "Treatment"],
    "Outcome": ["Converted", "Not Converted", "Converted", "Not Converted"],
    "Count": [120, 880, 150, 850]
}

df = pd.DataFrame(data)

# Create a contingency table
contingency_table = df.pivot(index="Group", columns="Outcome", values="Count")

# Run the chi-square test
chi2, p, dof, expected = chi2_contingency(contingency_table)

print(f"Chi-square statistic: {chi2:.2f}")
print(f"p-value: {p:.4f}")
print("Expected frequencies:")
print(pd.DataFrame(expected, index=contingency_table.index, columns=contingency_table.columns))