Summary  
This chapter covers how to implement statistical significance testing in code using p-values and t-tests to assess whether observed differences between two data samples are unlikely to be due to random chance.

General domain of usage  
A/B testing

**Video Description:**  
This video introduces the concept of **statistical significance** in A/B testing, using simple analogies like **coin flips** and **product test outcomes**. You'll learn how to tell if a result is likely due to your changes or just random chance, and why this matters when making product decisions. Cut all examples of code to a minimum. Show only the essential code. Show code no longer than 10 lines. Remove all possible pieces to demonstrate some code feature. You may show not full code that will not start, but still demonstrates some feature theoritacally.
If code is longer than 8 lines, move code output  blocks to second column.

When you run an A/B test, you want to know if the difference you see between the control and variant groups is real or just a result of random chance. Think about flipping a coin: if you flip it ten times and get seven heads, does that mean the coin is unfair? Or was it just luck? In product analytics, this is where **statistical significance** comes in. It helps you decide if the difference in outcomes - like more users clicking a new button - is likely to be meaningful, or if it could have happened just by accident, like a streak of heads in coin flips.

import numpy as np
from scipy import stats

# Simulated data: daily conversions for control and variant groups
control = np.array([30, 28, 35, 33, 29, 31, 32])
variant = np.array([36, 34, 39, 37, 35, 38, 40])

# Performing independent t-test
t_stat, p_value = stats.ttest_ind(variant, control)

print("t-statistic:", t_stat)
print("p-value:", p_value)

if p_value < 0.05:
    print("Result is statistically significant: the variant performed differently from control.")
else:
    print("Result is not statistically significant: no strong evidence of a difference.")

 **Statistical significance** indicates that observed differences are **unlikely due to random chance**.

Definition

When you get a **p-value** from your statistical test, it tells you how likely it is to see a difference as large - or larger - than the one you observed, just by chance. A **low p-value** (for example, below `0.05`) means it's unlikely the results happened randomly, so you can be more confident that your change made a real impact. If the p-value is high, you can't rule out that the difference was just luck. This helps you make product decisions with confidence: launch new features when the evidence is strong, and avoid acting on results that might not hold up.


The **significance level**, often shown as `α` (alpha), is a threshold you set before running your test to decide how much risk of a **false positive** (Type I error) you are willing to accept. In A/B testing, it represents the probability of incorrectly concluding that a real difference exists when, in fact, the difference is just due to random chance.

- The most common significance level is `0.05`, or 5%; 
- This means you accept a 5% chance of wrongly declaring a difference when there is none; 
- Lowering the significance level (for example, to `0.01`) makes your test more strict, reducing the risk of a false positive but requiring stronger evidence to declare significance; 
- The significance level is set before you collect or analyze your data.

In practice, if your **p-value** is less than your chosen significance level, you consider the result **statistically significant** and more likely to reflect a real effect. If the p-value is higher, you do not have enough evidence to confidently say there is a true difference. Setting the right significance level helps you balance the risks of making wrong decisions in your product experiments.

What does a low p-value indicate in hypothesis testing?

Explore the core metrics and frameworks product analysts use to measure user behavior, retention, and growth. This course covers the full product analytics workflow, from data collection and cleaning to advanced analysis and visualization techniques, empowering you to make data-driven product decisions.

Explore the foundational frameworks product analysts use to understand user behavior and business goals.

Dive into the essential metrics every product analyst must know, including engagement, retention, conversion, and churn, with hands-on calculation and interpretation.

Explore advanced segmentation and cohort analysis techniques that reveal how different user groups behave over time.

Design, run, and interpret A/B tests and other experiments to validate product decisions with data.

Statistical Significance

1. What does a low p-value indicate in hypothesis testing?

2. Fill in the blank: