Introduction to Statistical Methods

Hypothesis Testing — ANOVA, Chi-Square, Correlation and Regression

📅 Published: 2026-07-07

🎓 Level: postgraduate

👥 Audience: Postgraduate students in statistics and machine learning

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Hypothesis testing fundamentals — covered in Lecture 10 (Sampling, Sampling Distributions, and Estimation)
Normal distribution and Z-scores — covered in Lecture 8 (Bernoulli, Binomial, Poisson, and Normal Distributions)
Central Limit Theorem and sampling distributions — covered in Lecture 10
Confidence intervals and significance levels — covered in Lecture 10

Hypothesis Testing: ANOVA, Chi-Square, Correlation and Regression

This lecture completes your hypothesis testing toolkit. You already know how to test one mean (Z-test, t-test), compare two means, and handle one or two proportions. Now you add three powerful extensions: the chi-square test for categorical association, ANOVA for comparing three or more means, and covariance/correlation for measuring relationships between continuous variables. By the end, you will have a complete decision tree: means → Z/t, proportions → Z/chi-square, several proportions → chi-square, several means → ANOVA, and relationships → correlation and regression.

12.1 P-value Approach to Hypothesis Testing

Hook. You run a regression in Python. The summary spits out a column labelled P > |t|. A number like 0.032 sits there. Is it small enough? What is it even telling you? You already know how to reject a hypothesis by checking whether a test statistic crosses a critical value. The P-value gives you the same answer — but from the opposite direction. Instead of asking "is my statistic extreme?", you ask "how surprised should I be if the null is true?"

Intuition + Analogy. Imagine you claim a coin is fair. I flip it 100 times and get 62 heads. That feels like evidence against fairness. But how much evidence? The P-value is the probability of seeing 62 or more heads — or 62 or fewer tails on the other extreme — from a fair coin. If that probability is tiny (say, 0.01), you think: "A fair coin almost never does this. I don't believe the coin is fair." If the probability is large (say, 0.40), you think: "Fair coins do this all the time. Nothing to see here."

The P-value is your surprise meter. The smaller it is, the more surprised you are — and the less you trust the null hypothesis.

When the analogy breaks: The coin flip gives a two-sided P-value naturally. In practice you must decide whether your test is one-tailed or two-tailed. The P-value computation changes accordingly.

12.1.1 The Two Approaches to Decision Making

Formalize. Every hypothesis test boils down to the same structure. You have a null hypothesis $H_{0}$ (the default claim) and an alternative $H_{1}$ (what you suspect might be true instead). You gather data and compute a test statistic — a single number that summarizes the evidence. Then you have two ways to decide:

Approach 1 — Critical region. You pick a significance level $α$ (say, 0.05). You find the critical value that cuts off the outer $α$ of the null distribution. You check whether your test statistic lands in that rejection region.

Approach 2 — P-value. You compute your test statistic. You ask: "If $H_{0}$ were true, how often would I see a test statistic this extreme — or more?" That probability is the P-value. Then:

$ p \leq α p > α ⟹ Reject H_{0} ⟹ Do not reject H_{0} $

Both approaches always agree. A test statistic inside the critical region means $p \leq α$ . A test statistic outside means $p > α$ . They are two sides of the same coin.

Worked Example — Z-test with P-value. Suppose you run a Z-test on some data and get a test statistic $Z = 2.387$ . You chose $α = 0.05$ and this is a two-tailed test.

From the Z-table, the area to the right of $Z = 2.387$ is about 0.0085. Since this is two-tailed, the P-value is twice that: $p \approx 0.0170$ .

Now compare: $0.0170 \leq 0.05$ . Reject $H_{0}$ .

Sense-check: A Z of 2.387 is nearly 2.4 standard deviations from the mean. Values this far out are rare — about 1.7% of the time under the null. That is less than our 5% tolerance, so we reject. The conclusion makes sense.

12.1.2 Symbol Registry — P-value Approach

$α$ — level of significance — the threshold probability you set before seeing data (commonly 0.05, 0.01, or 0.10)
$p$ — P-value — the probability, assuming $H_{0}$ is true, of observing a test statistic at least as extreme as the one you computed
$H_{0}$ — null hypothesis — the default claim you are testing (e.g., "no effect," "no difference")
$H_{1}$ — alternative hypothesis — the claim you accept if you reject $H_{0}$

12.1.3 When to Use Each Approach

Scope. The critical-region approach shines in textbook problems — you are handed a table, you look up a number, you draw a line. The P-value approach is what every software package reports (R, Python statsmodels, SPSS, Excel). Why? Because the P-value gives you more information. Instead of a binary "reject / don't reject," you get a continuous measure of evidence strength. A P-value of 0.049 and 0.001 both say "reject at $α = 0.05$ ," but the second is far stronger evidence. The critical-region approach hides that gradient.

Both approaches assume you have correctly identified the test (Z, t, F, $χ^{2}$ ) and the tail type (one-tailed vs. two-tailed). Get either wrong, and both approaches give the wrong answer.

Visual Intuition. Picture the bell curve of the null distribution. Mark your test statistic on the horizontal axis. The P-value is the area under the curve beyond that mark — the tail probability. For a two-tailed test, it is the area in both tails combined. The smaller that shaded area, the further your statistic sits from the center, and the harder it is to believe $H_{0}$ is true. The $α$ threshold is a vertical line on the area scale: if the shaded region is smaller than $α$ -worth of area, you reject.

Pitfalls.

"The P-value is the probability that $H_{0}$ is true." Wrong. The P-value assumes $H_{0}$ is true. It tells you how unusual your data are under that assumption. It does not tell you the probability that $H_{0}$ is correct.
"A large P-value proves $H_{0}$ ." Wrong. A large P-value means the data are consistent with $H_{0}$ — not that $H_{0}$ is true. You might just have too little data to detect a real effect.
"P < 0.05 means the result is practically important." Wrong. Statistical significance is not the same as practical significance. With a huge sample, a tiny, meaningless effect can produce a tiny P-value.
One-tailed vs. two-tailed confusion. If you run a two-tailed test and report a one-tailed P-value (or vice versa), your conclusion flips. Always be explicit about which tail you are using.

Recap. The P-value is the probability of seeing data like yours (or more extreme) if the null were true. Compare it against $α$ : smaller → reject, larger → don't reject. It is the software era's answer to the critical-region approach — same logic, richer output. Bridge: The P-value appears in every test that follows — chi-square, ANOVA, and regression all report P-values. The rule $p \leq α \Rightarrow$ reject $H_{0}$ is universal.

Real-World & Domain Connection. Every regression table in economics, every clinical trial result in medicine, every A/B test dashboard in tech reports P-values. When you run model.fit() in scikit-learn and then inspect the summary via statsmodels, the P>|t| column tells you whether each coefficient is statistically different from zero. In ML, P-values help you decide which features to keep — and which might just be noise. But remember: a feature with a high P-value could still matter. The P-value does not replace thinking. It supplements it.

12.2 Chi-Square Test

Hook. You have a spreadsheet with 400 accident records. Each row says: "early hours? yes/no" and "fatal? yes/no." You stare at the numbers and wonder — are late-night accidents really more deadly, or is it just random noise? A Z-test can compare one proportion. A t-test compares means. But here you have several proportions tangled together in a table. You need a test built for categorical grids. That test is the chi-square ( $χ^{2}$ ) test.

Intuition + Analogy. Picture a jury pool. The court expects the pool to match the community's demographics: same proportions of age groups, genders, ethnicities. If the pool has 80% men when the community is 50% men, you get suspicious. The chi-square test does exactly this comparison — but for any number of categories. It asks: "If the two variables really were independent, what counts would I expect to see in each cell?" Then it measures how far the actual counts are from those expectations. The bigger the gap, the more evidence that the variables are linked.

Think of it like checking a restaurant bill. You ordered three dishes. The items and prices are listed. You multiply each item's quantity by its price. You sum them up. The total should match. If it does not, the bill has a problem. Chi-square works the same way: it computes an "expected bill" from the row and column totals. Then it checks whether the "observed bill" matches. The mismatch, squared and scaled, is $χ^{2}$ .

When the analogy breaks: The restaurant bill analogy captures the proportional-allocation logic. But chi-square does not test for errors in arithmetic — it tests whether the pattern of mismatch is too large to be chance. Also, chi-square works with counts, not prices — every cell must be a frequency.

12.2.1 Overview and When to Use

Formalize. The chi-square test handles categorical data. Where Z and t tests deal with means, and Z tests for proportions handle one or two groups, the chi-square test handles several proportions at once. It is classified as a non-parametric test — it does not assume your data come from a normal distribution or any specific distribution.

The chi-square ( $χ^{2}$ ) distribution arises from summing squared standard normal variables. If $Z_{1}, Z_{2}, \dots, Z_{k}$ are independent standard normal variables, then:

$ χ^{2} = Z_{1}^{2} + Z_{2}^{2} + \dots + Z_{k}^{2} $

follows a chi-square distribution with $k$ degrees of freedom. This is why the test statistic squares differences and divides by expectation — it turns count discrepancies into a quantity that, under $H_{0}$ , follows a known distribution.

There are two main types of chi-square tests:

Chi-square test for independence — tests whether two categorical variables are associated or independent. Example: "Is smoking habit related to hypertension?"
Chi-square test for goodness of fit — tests whether observed frequency data follows a particular theoretical distribution (binomial, Poisson, normal, etc.).

12.2.2 Chi-Square Test for Independence

Purpose. You have data organized in a contingency table — rows are one categorical variable, columns are another. Each cell contains a count (frequency). You want to know: are these two variables related, or not?

Hypotheses. The convention is always the same:

$H_{0}$ : The two variables are independent (no association).
$H_{1}$ : The two variables are not independent (they are associated).

Never swap these. Putting "they are associated" as $H_{0}$ leads to logical confusion. The test is built to detect departures from independence — so independence is the null.

12.2.3 Chi-Square Test for Goodness of Fit

Purpose. You have frequency data and you suspect it follows a known distribution — binomial, Poisson, normal, etc. The goodness-of-fit test lets you check this formally.

Heuristic. A practical shortcut: if the sample mean is larger than the sample variance, the binomial distribution might be a reasonable fit. The chi-square goodness-of-fit test gives you a rigorous yes/no instead of a heuristic guess.

How it works: You compute expected frequencies under the hypothesized distribution. Then you apply the same $χ^{2}$ formula: sum of $(O - E)^{2} / E$ across all categories. The degrees of freedom are $(number of categories - 1 - number of estimated parameters)$ . Compare against the chi-square table.

12.2.4 Formula and Computation

The chi-square statistic:

$ χ^{2} = i = 1 \sum R j = 1 \sum C \frac{( O _{ij} - E _{ij} ) ^{2}}{E _{ij}} $

Symbol Registry — Chi-Square Test

$O_{ij}$ — observed frequency for the cell in row $i$ , column $j$ — the actual count from your data
$E_{ij}$ — expected frequency for that same cell — the theoretical count if $H_{0}$ (independence) were true
$R$ — number of rows (do not count the "Total" row)
$C$ — number of columns (do not count the "Total" column)
Degrees of freedom — $(R - 1) \times (C - 1)$

How to compute expected frequencies. Under independence, the proportion of observations in row $i$ should be the same across all columns. So:

$ E_{ij} = \frac{( Row i total ) \times ( Column j total )}{Grand total} $

This is proportional allocation. The grand total is spread across cells in proportion to the row and column shares. This is exactly what you would expect if the two classification variables had nothing to do with each other.

Decision rule: Compare the computed $χ^{2}$ to the critical value from the chi-square table at your chosen $α$ and degrees of freedom $(R - 1) (C - 1)$ . If computed $χ^{2}$ is greater than the critical value, reject $H_{0}$ .

The chi-square distribution is not symmetric like Z or t. It is right-skewed, and the skew lessens as degrees of freedom increase. The rejection region is always on the right tail. When your computed $χ^{2}$ exceeds the critical value, it falls in the rejection region — the observed counts are too far from expected to be chance.

12.2.5 Worked Example 1 — Accident Timing and Fatality

Problem: A study of 400 accident cases produced the following data. Does the data indicate any association between the time of the accident and its fatality? Test at $α = 0.05$ .

Observed Data:

	Early Hours (Yes)	Early Hours (No)	Total
Fatal	80	19	99
Non-Fatal	200	101	301
Total	280	120	400

Step 1 — Hypotheses:

$H_{0}$ : Time of accident and fatality are independent.
$H_{1}$ : Time of accident and fatality are not independent (they are associated).

Step 2 — Compute expected frequencies. For each cell, $E = (row total \times col total) / grand total$ :

Fatal, Early Hours Yes: $E = \frac{280 \times 99}{400} = \frac{27720}{400} = 69.3$
Fatal, Early Hours No: $E = \frac{120 \times 99}{400} = \frac{11880}{400} = 29.7$
Non-Fatal, Early Hours Yes: $E = \frac{280 \times 301}{400} = \frac{84280}{400} = 210.7$
Non-Fatal, Early Hours No: $E = \frac{120 \times 301}{400} = \frac{36120}{400} = 90.3$

Step 3 — Compute $χ^{2}$ . For each cell, $(O - E)^{2} / E$ :

$ Fatal, Yes: Fatal, No: Non-Fatal, Yes: Non-Fatal, No: (80 - 69.3)^{2} /69.3 = (10.7)^{2} /69.3 = 114.49/69.3 \approx 1.652 (19 - 29.7)^{2} /29.7 = (- 10.7)^{2} /29.7 = 114.49/29.7 \approx 3.855 (200 - 210.7)^{2} /210.7 = (- 10.7)^{2} /210.7 = 114.49/210.7 \approx 0.543 (101 - 90.3)^{2} /90.3 = (10.7)^{2} /90.3 = 114.49/90.3 \approx 1.268 $

$ χ^{2} \approx 1.652 + 3.855 + 0.543 + 1.268 = 7.318 $

Step 4 — Critical value. Degrees of freedom = $(2 - 1) \times (2 - 1) = 1$ . From the chi-square table at $α = 0.05$ and df = 1: $χ_{critical}^{2} = 3.841$ .

Step 5 — Decision. $7.318 > 3.841$ . The test statistic falls in the rejection region. Reject $H_{0}$ .

Conclusion: There is a significant association between the time of the accident and its fatality. The two variables are not independent.

Sense-check: In the observed data, 80 out of 99 fatal accidents (about 81%) happened in early hours, versus 200 out of 301 non-fatal accidents (about 66%). The gap is large enough that random chance cannot explain it. The test confirms what the raw percentages suggest.

12.2.6 Worked Example 2 — Smoking Habits and Hypertension

Problem: A study of 180 individuals collected data on smoking habits (non-smoker, moderate smoker, heavy smoker) and whether they have hypertension. Test whether smoking habit and hypertension are associated.

Observed Data:

	Non-Smoker	Moderate Smoker	Heavy Smoker	Total
Hypertension (Yes)	21	36	30	87
Hypertension (No)	48	26	19	93
Total	69	62	49	180

Step 1 — Hypotheses:

$H_{0}$ : Smoking habit and hypertension are independent.
$H_{1}$ : Smoking habit and hypertension are not independent.

Step 2 — Compute expected frequencies for all six cells:

$ E_{11} E_{21} = \frac{69 \times 87}{180} = 33.35 = \frac{69 \times 93}{180} = 35.65 E_{12} E_{22} = \frac{62 \times 87}{180} = 29.97 = \frac{62 \times 93}{180} = 32.03 E_{13} E_{23} = \frac{49 \times 87}{180} = 23.68 = \frac{49 \times 93}{180} = 25.32 $

Step 3 — Compute $χ^{2}$ for each cell, $(O - E)^{2} / E$ :

$ (Yes, Non): (Yes, Mod): (Yes, Heavy): (No, Non): (No, Mod): (No, Heavy): (21 - 33.35)^{2} /33.35 = (- 12.35)^{2} /33.35 = 152.52/33.35 \approx 4.574 (36 - 29.97)^{2} /29.97 = (6.03)^{2} /29.97 = 36.36/29.97 \approx 1.213 (30 - 23.68)^{2} /23.68 = (6.32)^{2} /23.68 = 39.94/23.68 \approx 1.687 (48 - 35.65)^{2} /35.65 = (12.35)^{2} /35.65 = 152.52/35.65 \approx 4.278 (26 - 32.03)^{2} /32.03 = (- 6.03)^{2} /32.03 = 36.36/32.03 \approx 1.135 (19 - 25.32)^{2} /25.32 = (- 6.32)^{2} /25.32 = 39.94/25.32 \approx 1.578 $

$ χ^{2} \approx 4.574 + 1.213 + 1.687 + 4.278 + 1.135 + 1.578 = 14.465 $

Step 4 — Critical value. $R = 2$ rows, $C = 3$ columns. df = $(2 - 1) \times (3 - 1) = 2$ . At $α = 0.05$ and df = 2: $χ_{critical}^{2} = 5.991$ .

Step 5 — Decision. $14.465 > 5.991$ . Reject $H_{0}$ .

Conclusion: Smoking habit and hypertension are significantly associated.

Sense-check: Look at the row percentages. Among those with hypertension: 24% non-smokers, 41% moderate, 34% heavy. Among those without: 52% non-smokers, 28% moderate, 20% heavy. The pattern reverses — non-smokers dominate the no-hypertension group while moderate/heavy smokers dominate the hypertension group. The test confirms this is not random noise.

12.2.7 Student Questions and Answers — Chi-Square

Q: How was the chi-square formula derived? Is there a logic behind it?

A: The chi-square distribution comes from summing squared standard normal variables: if $Z_{1}, \dots, Z_{k} \sim N (0, 1)$ are independent, then $\sum Z_{i}^{2} \sim χ_{k}^{2}$ . The test statistic $\sum (O - E)^{2} / E$ approximates this sum because each cell's standardized residual $(O - E) / E$ behaves roughly like a standard normal under $H_{0}$ when sample sizes are large. The full derivation involves the multinomial distribution and a Taylor expansion — but for application purposes, knowing how to compute and interpret the statistic is what matters.

Q: How do I know when to use chi-square versus a t-test or Z-test?

A: Use the decision tree. Means (one mean, two means) → Z or t. Proportions (one proportion, two proportions) → Z. Several proportions, or testing association between categorical variables → chi-square. Chi-square is non-parametric — it makes no normality assumption. The key question: am I comparing counts in categories, or am I comparing numerical averages? Counts → chi-square. Averages → Z/t/ANOVA.

Q: Can we exchange $H_{0}$ and $H_{1}$ ? Make the hypothesis of association the null?

A: No. The convention in chi-square is fixed: $H_{0}$ says "the two variables are independent," and $H_{1}$ says "they are not independent." This is the standard framing for a test of independence. Swapping them leads to confusion about what the test is measuring. The chi-square statistic is built to measure departures from independence — so independence must be the baseline ( $H_{0}$ ).

Q: What is the ideal chi-square value?

A: There is no "ideal" value. The statistic is computed from data and compared against the critical value from the table. The critical value depends on your chosen $α$ and degrees of freedom. A small $χ^{2}$ means the data fits the independence assumption well. A large $χ^{2}$ means it doesn't. The decision (reject or not) comes from the comparison, not from the raw number. Think of it like a thermometer — 98.6°F is "normal" but the number itself has no moral weight.

Q: How do we arrive at the degrees of freedom? Why $(R - 1) (C - 1)$ ?

A: Degrees of freedom for chi-square independence is $(R - 1) \times (C - 1)$ . Here is why. In a contingency table with fixed row and column totals, once you fill $(R - 1)$ rows and $(C - 1)$ columns, the remaining cells are determined — they are not free to vary. So you have $(R - 1) (C - 1)$ independent cells. For a $2 \times 2$ table: $(2 - 1) (2 - 1) = 1$ . For a $2 \times 3$ table: $(2 - 1) (3 - 1) = 2$ .

Q: What if $α$ is not given in the problem?

A: You are free to choose. Common choices are 1% (0.01), 5% (0.05), or 10% (0.10). State your choice explicitly: "I am testing at a 5% level of significance." Be aware that the conclusion can change with a different $α$ . A result significant at 5% may not be significant at 1%. This is normal — different confidence thresholds can lead to different decisions.

Q: How do we know the rejection region for chi-square? Is it right-tailed?

A: The chi-square distribution is right-skewed and always positive, so the concept of "left-tailed" does not apply naturally. The rejection region is on the right side. If computed $χ^{2}$ is greater than the critical value, reject $H_{0}$ . If less, do not reject. For example, at df = 1 and $α = 0.05$ : critical = 3.841. A computed value of 7.31 → reject. A computed value of 2.14 → do not reject. The logic: larger $χ^{2}$ means larger mismatch between observed and expected — exactly what you would see if the variables were not independent.

Q: Is chi-square only for two proportions? The smoking example has three categories.

A: No. Chi-square handles several proportions. The smoking example involves multiple proportions simultaneously: 87/180 have hypertension, 69/180 are non-smokers, 21/69 non-smokers have hypertension, and so on. All these proportions collectively test whether the variables are independent. This is why chi-square sits at the "several proportions" branch of the decision tree.

Q: Is a contingency table similar to a confusion matrix?

A: Structurally, yes — both are grids of counts with rows and columns. But a confusion matrix serves a different purpose: it compares predicted labels against true labels in classification. The chi-square test asks a different question: are the row and column classifications independent? The structural resemblance is helpful for recognizing contingency tables, but chi-square has its own statistical framework. Don't mix up the two contexts.

Scope. The chi-square test for independence assumes:

Random sampling. The data come from a random sample.
Independent observations. Each subject contributes to exactly one cell.
Adequate expected frequencies. A common rule of thumb: no expected cell frequency should be below 1, and no more than 20% of cells should have expected frequencies below 5. If this rule is violated, the $χ^{2}$ approximation to the true sampling distribution deteriorates. For $2 \times 2$ tables with small samples, use Fisher's exact test instead.
Categorical data. Both variables must be categorical (nominal or ordinal). Chi-square does not work with continuous variables unless they are binned — and binning loses information.

Visual Intuition. Imagine a $2 \times 2$ grid of four bars. The height of each bar is the observed count. Superimpose a second set of bars — shorter or taller — representing the expected counts under independence. The chi-square statistic is the sum, over all four bars, of (squared height difference) divided by expected height. A bar where observed is 80 and expected is 69 contributes $(11)^{2} /69 \approx 1.75$ . A bar where observed is 101 and expected is 90 contributes $(11)^{2} /90 \approx 1.34$ . The final $χ^{2}$ is the total squared-and-scaled mismatch across the whole grid.

Pitfalls.

Small expected frequencies. If any $E_{ij}$ is too small (below 5, and especially below 1), the chi-square approximation breaks down. The test may give a significant result when it shouldn't, or fail to detect a real association. Always check the expected frequency table before trusting the result.
Chi-square does not measure strength of association. A significant result tells you the variables are probably related. It does not tell you how strongly. For that, use measures like Cramér's V or the phi coefficient.
Causation is not implied. "Smoking is associated with hypertension" does not mean "smoking causes hypertension." Chi-square only detects association, not causation.
The test is sensitive to sample size. With a huge sample, even a very small association becomes statistically significant. With a tiny sample, a large real association might be missed. Always report effect size alongside the P-value.
Don't confuse goodness-of-fit with independence. They use the same formula but answer different questions. Independence = two variables in a contingency table. Goodness-of-fit = one variable compared to a theoretical distribution.

Recap. The chi-square test compares observed counts to expected counts under the assumption of independence. $χ^{2} = \sum (O - E)^{2} / E$ . If the computed value exceeds the critical value at df = $(R - 1) (C - 1)$ , reject $H_{0}$ and conclude the variables are associated. Bridge: Chi-square handles several proportions. The next test — ANOVA — handles several means, using a similar "compare two kinds of variation" logic.

Real-World & Domain Connection. Chi-square tests appear everywhere categorical decisions are made. Market researchers use them to see if purchase intent differs by age group. Medical researchers test whether a treatment response varies by genetic marker. In A/B testing, chi-square tests whether conversion rates differ across multiple landing page variants. In ML, chi-square is used as a feature selection method — sklearn.feature_selection.chi2 scores each feature against the target to identify the most relevant categorical predictors. The chi-square test of independence is one of the oldest statistical tools still in wide use. Karl Pearson invented it in 1900, and it has never gone out of style.

12.3 ANOVA — Analysis of Variance

Hook. You are testing three different fertilizers on crop yield. Fertilizer A averages 10 bushels, B averages 8, C averages 12. Are these differences real, or just random variation? A t-test compares two means. But you have three. You could run three separate t-tests — A vs B, B vs C, A vs C — but each test carries its own risk of a false positive. Run enough t-tests and you are practically guaranteed to "find" something that isn't there. You need a single test that asks: "Are all these means the same?" That test is ANOVA.

Intuition + Analogy. Imagine three archers shooting at their own targets. Each archer has a spread (their shots cluster around their own bullseye). The three bullseyes might also be at different positions. ANOVA separates the total variation into two buckets:

Within-group variation: How much does each archer's own spread vary? This is natural noise — shooting inconsistency.
Between-group variation: How far apart are the three bullseyes? If the bullseyes are far apart relative to each archer's spread, you conclude the archers are aiming at different targets.

The F-statistic is the ratio: between-group variation divided by within-group variation. A large F means the bullseyes are far apart compared to the spread — the group means are genuinely different. A small F means the bullseyes might as well be in the same place — any apparent differences could just be random wobble.

When the analogy breaks: Real data can have unequal group sizes and unequal within-group variances. The basic ANOVA assumes equal variances (homoscedasticity) and normality within each group. When these break, you need Welch's ANOVA or non-parametric alternatives like the Kruskal-Wallis test.

12.3.1 Overview and When to Use

Formalize. ANOVA stands for ANalysis Of VAriance. The name is honest: although the goal is to compare means, the method works by analyzing variances. Specifically, it partitions the total variation in the data into components attributable to different sources.

The decision tree:

One mean → single-mean Z or t test
Two means → two-sample Z or t test
Three or more means → ANOVA (F-test)

Just as chi-square handles several proportions, ANOVA handles several means. Both use a single test to avoid the multiple-comparisons problem — the inflation of Type I error that comes from running many pairwise tests.

The null hypothesis in ANOVA is always:

$ H_{0} : μ_{1} = μ_{2} = μ_{3} = \dots = μ_{k} $

The alternative:

$ H_{1} : At least one μ_{i} differs from the others $

Note: $H_{1}$ does not say all means differ. It says at least one differs. ANOVA tells you there is a difference somewhere. It does not tell you which groups differ. For that, you need post-hoc tests (Tukey's HSD, Bonferroni, etc.).

12.3.2 The Two Kinds of Variance

Think about what happens when you have $k$ groups (samples from $k$ populations). There are two sources of variation:

Between-group variance (treatment variance). How different are the group means from one another? If the population means truly differ, the group means will be spread out. This is captured by the Sum of Squares due to Treatment (SSTR) — also called SSB (Sum of Squares Between).
Within-group variance (error variance). How much do individual observations vary within each group? This is the natural noise — measurement error, individual differences, uncontrolled factors. It is captured by the Sum of Squares due to Error (SSE) — also called SSW (Sum of Squares Within).

The F-statistic is the ratio:

$ F = \frac{M S T R}{M S E} = \frac{SSTR / ( k - 1 )}{SSE / ( n - k )} $

MSTR is the Mean Square for Treatment — the average between-group variation per degree of freedom. MSE is the Mean Square for Error — the average within-group variation per degree of freedom.

If the between-group variation is large relative to the within-group variation, F is large, and you reject $H_{0}$ . If the groups overlap heavily (within-group spread dominates), F is close to 1 or less, and you fail to reject $H_{0}$ .

12.3.3 The ANOVA Computation Framework

Symbol Registry — One-Way ANOVA

$k$ — number of groups (populations being compared)
$n_{j}$ — number of observations in group $j$
$n = \sum n_{j}$ — total number of observations across all groups
$X ˉ$ — grand mean (mean of all observations combined)
$X ˉ_{j}$ — mean of group $j$
$X_{ij}$ — the $i$ -th observation in group $j$
SSTR — Sum of Squares due to Treatment (between groups)
SSE — Sum of Squares due to Error (within groups)
SST — Total Sum of Squares = SSTR + SSE
MSTR — Mean Square Treatment = SSTR / $(k - 1)$
MSE — Mean Square Error = SSE / $(n - k)$
$F$ — test statistic = MSTR / MSE
Degrees of freedom: numerator = $k - 1$ , denominator = $n - k$

SSTR formula. For each group, take the group mean's deviation from the grand mean. Square it. Multiply by the number of observations in that group. Sum across all groups:

$ S S T R = j = 1 \sum k n_{j} (X ˉ_{j} - X ˉ)^{2} $

SSE formula. For each group, take each observation within that group, find its deviation from that group's mean, square it, and sum across all observations:

$ S S E = j = 1 \sum k i = 1 \sum n_{j} (X_{ij} - X ˉ_{j})^{2} $

The partition identity. The total sum of squares always decomposes as:

$ S S T = S S T R + S S E $

where $S S T = \sum_{j = 1}^{k} \sum_{i = 1}^{n_{j}} (X_{ij} - X ˉ)^{2}$ . This identity says: total deviation from the grand mean equals the deviation explained by group membership plus the unexplained deviation within groups. It is the same logic as $R^{2}$ in regression — and in fact, ANOVA is a special case of linear regression with categorical predictors.

12.3.4 Worked Example — Three Groups

Problem: Three groups with five observations each. Test at $α = 0.05$ whether there is a significant difference in the means of the three populations (assume normal populations with equal variances).

Group 1	Group 2	Group 3
8	7	12
10	5	9
10	8	12
10	10	15
12	10	12

Step 1 — Group means and grand mean:

$ X ˉ_{1} = \frac{8 + 10 + 10 + 10 + 12}{5} = 10, X ˉ_{2} = \frac{7 + 5 + 8 + 10 + 10}{5} = 8, X ˉ_{3} = \frac{12 + 9 + 12 + 15 + 12}{5} = 12 $

$ X ˉ = \frac{10 + 8 + 12}{3} = 10 $

Step 2 — SSTR (Between-groups):

$ S S T R = 5 (10 - 10)^{2} + 5 (8 - 10)^{2} + 5 (12 - 10)^{2} = 5 (0) + 5 (4) + 5 (4) = 0 + 20 + 20 = 40 $

Step 3 — SSE (Within-groups):

Group 1: $(8 - 10)^{2} + (10 - 10)^{2} + (10 - 10)^{2} + (10 - 10)^{2} + (12 - 10)^{2} = 4 + 0 + 0 + 0 + 4 = 8$

Group 2: $(7 - 8)^{2} + (5 - 8)^{2} + (8 - 8)^{2} + (10 - 8)^{2} + (10 - 8)^{2} = 1 + 9 + 0 + 4 + 4 = 18$

Group 3: $(12 - 12)^{2} + (9 - 12)^{2} + (12 - 12)^{2} + (15 - 12)^{2} + (12 - 12)^{2} = 0 + 9 + 0 + 9 + 0 = 18$

$ S S E = 8 + 18 + 18 = 44 $

Step 4 — SST check: $S S T = S S T R + S S E = 40 + 44 = 84$ . ✓

Step 5 — Degrees of freedom:

SSTR df: $k - 1 = 3 - 1 = 2$
SSE df: $n - k = 15 - 3 = 12$

Step 6 — Mean squares:

$ M S T R = \frac{40}{2} = 20, M S E = \frac{44}{12} \approx 3.667 $

Step 7 — F-statistic:

$ F = \frac{20}{3.667} \approx 5.45 $

Step 8 — Critical value. From the F-table at $α = 0.05$ , with numerator df = 2 and denominator df = 12: $F_{critical} = 3.89$ .

Step 9 — Decision. $5.45 > 3.89$ . Reject $H_{0}$ . There is a significant difference among the three group means.

Sense-check: Group 2's mean (8) is notably lower than Group 1 (10) and Group 3 (12). The within-group spreads are moderate (SSE = 44). The between-group spread (SSTR = 40) is large enough relative to the within-group noise to conclude these differences are real. The F of 5.45 comfortably clears the 3.89 threshold.

12.3.5 Understanding the F-Table

The F-distribution has two degrees of freedom — one for the numerator ( $ν_{1} = k - 1$ ) and one for the denominator ( $ν_{2} = n - k$ ). The F-table is two-dimensional:

The numerator degrees of freedom ( $ν_{1}$ ) usually runs across the top row as column headers.
The denominator degrees of freedom ( $ν_{2}$ ) usually runs down the first column as row labels.
The body of the table gives F critical values for a specific $α$ (typically separate tables for 0.05 and 0.01).

To look up a critical value: first pick the table with your chosen $α$ . Then find the column for your numerator df. Then find the row for your denominator df. The number at that intersection is your critical F.

The F-distribution is right-skewed and always positive (it is a ratio of variances, which are always non-negative). The rejection region is always on the right tail. A larger F means stronger evidence against $H_{0}$ .

12.3.6 Two-Way ANOVA

Comparison: One-Way vs. Two-Way ANOVA. Sometimes you have two independent classification variables. For example: you are testing three kinds of fertilizers (Factor A: treatment) across four different soil types (Factor B: blocks). You want to know if either factor affects yield.

Feature	One-Way ANOVA	Two-Way ANOVA
Number of factors	1	2
Hypotheses tested	1 (are all group means equal?)	2 (is Factor A significant? is Factor B significant?)
SS components	SSTR + SSE = SST	SSTR + SSBL + SSE = SST
F-statistics computed	1 (MSTR/MSE)	2 ( $F_{A}$ and $F_{B}$ )
When to use	One categorical predictor	Two categorical predictors simultaneously

In two-way ANOVA, you compute two F-statistics:

$ F_{A} = \frac{M S T R}{M S E} (tests Factor A), F_{B} = \frac{M S B L}{M S E} (tests Factor B) $

Each F has its own degrees of freedom: $(k_{A} - 1, error df)$ for Factor A, and $(k_{B} - 1, error df)$ for Factor B. Compare each against its respective critical value from the F-table. Make two independent decisions — one for each factor.

The Correction Factor (CF) sometimes appears in computational formulas. It is:

$ C F = \frac{G ^{2}}{n} $

where $G$ is the grand total of all observations (sum, not mean) and $n$ is the total number of observations. CF simplifies the sums-of-squares arithmetic by pre-computing the "baseline" term that appears in every SS formula.

12.3.7 ANOVA in the Machine Learning Context

This is where ANOVA becomes directly relevant to your ML workflow. When you train a multiple regression model:

$ y = w_{0} + w_{1} x_{1} + w_{2} x_{2} + w_{3} x_{3} + \dots $

The model summary (from statsmodels or R's lm) shows an F-statistic and its P-value. The hypothesis being tested is:

$ H_{0} : w_{1} = w_{2} = w_{3} = \dots = 0 $

$ H_{1} : At least one w_{i} \neq = 0 $

This is an ANOVA problem — testing whether any of several coefficients differ from zero. The F-test tells you whether the overall regression model explains a significant amount of variance in the response. If the F-statistic is large and the P-value is small, at least one predictor has a genuine relationship with the target.

This is the overall F-test for model significance — one of the most important diagnostics you will read in any regression output.

Note: ANOVA is computationally heavy. In exams, you will typically not be asked to solve a full ANOVA numerical from scratch. The emphasis is on understanding the concept, knowing when to apply it, and interpreting results. Assignments may include ANOVA computations.

12.3.8 Student Questions and Answers — ANOVA

Q: In ANOVA, is $H_{0}$ always about means being equal, unlike chi-square where $H_{0}$ is about independence?

A: Yes. The null hypothesis changes with the test because each test is designed to answer a different kind of question. In ANOVA, $H_{0}$ is that all population means are equal: $μ_{1} = μ_{2} = \dots = μ_{k}$ . In chi-square, $H_{0}$ is that the variables are independent. In a single-mean Z-test, $H_{0}$ is $μ = some specific value$ . Each test type has its own standard $H_{0}$ formulation tied to what the test evaluates.

Q: Can ANOVA be used to choose batch sizes in stochastic gradient descent — checking whether different groups have the same variance?

A: That is an interesting angle, but it is not a standard application. The F-test in ANOVA assumes normality and equal variances (homoscedasticity). In the SGD batch context, the gradient variance across batches depends on the data distribution and mini-batch composition in complex ways. Levene's test or Bartlett's test — which directly test equality of variances — would be more appropriate for comparing batch variances. ANOVA compares means, not variances, despite its name.

Q: What does $k$ stand for in the ANOVA formulas?

A: $k$ is the number of groups or populations being compared. $n$ is the total number of observations across all groups. Individual group sizes are $n_{j}$ , so $n = \sum_{j = 1}^{k} n_{j}$ .

Q: How do we read the F-table when there are two degrees of freedom?

A: The F-table is two-dimensional. The numerator degrees of freedom ( $ν_{1}$ ) is typically the column index across the top. The denominator degrees of freedom ( $ν_{2}$ ) is the row index down the side. To look up: identify the table for your $α$ (0.05, 0.01, etc.). Find your $ν_{1}$ column. Find your $ν_{2}$ row. The number at that intersection is your critical F-value. Always double-check which axis is which — different textbooks flip them.

Q: What is CF in the ANOVA calculation?

A: CF stands for Correction Factor. It equals $G^{2} / n$ , where $G$ is the grand total (sum of all observations, not the mean). It appears in the shortcut formulas for sums of squares. For example, $S S T = \sum\sum X_{ij}^{2} - C F$ . It simplifies the arithmetic by pre-computing a term that would otherwise be subtracted repeatedly.

Scope. One-way ANOVA assumes:

Independence. Observations within and between groups are independent.
Normality. Each group's population is roughly normally distributed. ANOVA is fairly resistant to moderate violations of normality, especially with equal sample sizes.
Homogeneity of variance (homoscedasticity). All groups have equal population variances. This is the most critical assumption. If variances differ substantially (especially with unequal sample sizes), use Welch's ANOVA instead.
Random sampling. Data are random samples from their respective populations.

What breaks when assumptions fail:

Unequal variances with unequal sample sizes → inflated Type I error (you reject $H_{0}$ too often).
Severe non-normality with small samples → the F-distribution approximation degrades.
Dependent observations (e.g., repeated measures on the same subjects) → use repeated-measures ANOVA instead.

Visual Intuition. Draw three bell curves side by side on the same axis, representing the three group distributions. The horizontal axis is the measurement scale. Each curve is centered at its group mean. The width of each curve reflects the within-group standard deviation. If the three curves overlap heavily — their peaks are close and their tails intermingle — you cannot tell them apart. The F is small. If the three curves are well-separated — their peaks are far apart relative to their widths — you can easily distinguish the groups. The F is large. The F-statistic quantifies this visual judgment: separation between peaks divided by average width.

Pitfalls.

ANOVA tells you that groups differ, not which groups differ. A significant F means at least one mean is different. To find which pairs differ, run post-hoc tests (Tukey's HSD is the most common). Don't just run pairwise t-tests — that inflates the familywise error rate.
Don't use ANOVA for two groups. A two-sample t-test and a one-way ANOVA with $k = 2$ give the same P-value (the F is exactly $t^{2}$ ). But the t-test is simpler and can be one-tailed. Use the right tool for the job size.
"Analysis of Variance" does not mean you are testing variances. You are testing means. The name refers to the method, not the hypothesis.
A non-significant F does not mean all means are equal. It means you lack evidence to conclude they differ. The sample sizes might be too small. Absence of evidence is not evidence of absence.
Confusing SSTR and SSE. SSTR = between-group variation (the signal you want to detect). SSE = within-group variation (the noise). If you swap them, your F is inverted and your conclusion is nonsense.

Recap. ANOVA compares three or more means by partitioning total variation into between-group (SSTR) and within-group (SSE) components. $F = M S T R / M S E$ . If F exceeds the critical value from the F-table at df = $(k - 1, n - k)$ , reject $H_{0}$ — at least one group mean differs. Bridge: ANOVA uses the F-test. The next concepts — covariance and correlation — shift focus from comparing groups to measuring relationships between two continuous variables.

Real-World & Domain Connection. ANOVA is the workhorse of experimental design. Pharmaceutical companies use it to compare drug dosages. Manufacturing engineers use it to compare production line outputs. Agricultural scientists use it to compare crop varieties (this is where ANOVA was born — R.A. Fisher developed it at the Rothamsted agricultural research station in the 1920s). In ML, the overall F-test in regression summaries is ANOVA. Every time you run statsmodels.api.OLS and look at the top of the summary output for "F-statistic" and "Prob (F-statistic)," you are reading an ANOVA result. In tech, A/B tests with more than two variants (A/B/C/D tests) use ANOVA to determine whether any variant outperforms the others before running pairwise comparisons.

12.4 Covariance

Hook. You have two columns of numbers — say, age and weight for five children. As age goes up, weight goes up. You can see it in the table. But how do you put a single number on that "moving together" pattern? Before you can measure the strength of a relationship, you need to measure its direction. Covariance is that first number.

Intuition + Analogy. Think of two dance partners. When one steps forward, the other also steps forward — that is positive covariance. When one steps forward and the other steps back — that is negative covariance. When their steps have no pattern at all — that is zero covariance.

Now, here is the math picture. Take each data point and ask: "Is X above or below its average? Is Y above or below its average?" Multiply those two deviations. If both are above-average (positive × positive = positive) or both below-average (negative × negative = positive), the product is positive — the pair agrees in direction. If one is above and the other below (positive × negative = negative), the product is negative — the pair disagrees. Sum all these products. Divide by $n - 1$ . That is covariance.

When the analogy breaks: The dance partners analogy captures direction only. It says nothing about how tightly they move together. A couple doing a formal waltz (tightly synchronized) and a couple casually swaying (loosely synchronized) could have the same direction but different strength. Covariance cannot tell you the difference — its magnitude is scale-dependent.

12.4.1 Definition and Formula

Formalize. Covariance measures how two variables move together. It is the expected value of the product of their deviations from their respective means:

$ Cov (X, Y) = \frac{\sum _{i = 1}^{n} ( X _{i} - X ˉ ) ( Y _{i} - Y ˉ )}{n - 1} $

Symbol Registry — Covariance

$X, Y$ — two random variables (or two columns of data)
$X ˉ$ — sample mean of $X$
$Y ˉ$ — sample mean of $Y$
$n$ — number of observations (use $n - 1$ for sample covariance, $n$ for population covariance)

A key identity. If you set $Y = X$ , then:

$ Cov (X, X) = \frac{\sum ( X _{i} - X ˉ ) ( X _{i} - X ˉ )}{n - 1} = \frac{\sum ( X _{i} - X ˉ ) ^{2}}{n - 1} = Var (X) $

Variance is just covariance of a variable with itself. This is why covariance is the more fundamental concept — variance is a special case.

12.4.2 Worked Example — Age and Weight

Data:

Age (X)	Weight (Y)
1	8
3	14
5	18
7	24
9	31

Step 1 — Means:

$ X ˉ = \frac{1 + 3 + 5 + 7 + 9}{5} = \frac{25}{5} = 5, Y ˉ = \frac{8 + 14 + 18 + 24 + 31}{5} = \frac{95}{5} = 19 $

Step 2 — Deviations and products:

$X_{i} - X ˉ$	$Y_{i} - Y ˉ$	Product
$1 - 5 = - 4$	$8 - 19 = - 11$	$(- 4) (- 11) = 44$
$3 - 5 = - 2$	$14 - 19 = - 5$	$(- 2) (- 5) = 10$
$5 - 5 = 0$	$18 - 19 = - 1$	$(0) (- 1) = 0$
$7 - 5 = 2$	$24 - 19 = 5$	$(2) (5) = 10$
$9 - 5 = 4$	$31 - 19 = 12$	$(4) (12) = 48$

Sum of products = $44 + 10 + 0 + 10 + 48 = 112$ .

Step 3 — Covariance (sample):

$ Cov (X, Y) = \frac{112}{5 - 1} = \frac{112}{4} = 28 $

(Some formulations divide by $n$ instead of $n - 1$ , giving $112/5 = 22.4$ . The $n - 1$ version is the sample covariance — the unbiased estimator of the population covariance.)

Sense-check: Age and weight both increase. Every product term is non-negative. The covariance is positive — that matches the visible pattern in the data.

12.4.3 Interpretation of Covariance

Positive covariance ( $Cov > 0$ ): X and Y tend to move in the same direction. When X is above its mean, Y tends to be above its mean.
Negative covariance ( $Cov < 0$ ): X and Y tend to move in opposite directions. When X is above its mean, Y tends to be below its mean.
Zero covariance ( $Cov \approx 0$ ): No consistent linear pattern of co-movement.

The critical limitation. Only the sign of covariance is interpretable. The magnitude is meaningless on its own. A covariance of 28 versus 280 tells you nothing about relationship strength. If you change units — pounds to kilograms, inches to centimeters — the covariance changes by a multiplicative factor. The number 28 is tied to the specific units of age (years) and weight (kg, or whatever units were used). Change the units and you change the covariance. There is no upper or lower bound — covariance can be any real number.

This is why covariance gives you direction but not strength. For strength, you need to normalize. That normalization is the correlation coefficient.

Visual Intuition. Plot the five (age, weight) points on a scatterplot. Draw a vertical line at $X ˉ = 5$ and a horizontal line at $Y ˉ = 19$ . These lines divide the plot into four quadrants. Points in the top-right quadrant (both above average) and bottom-left quadrant (both below average) contribute positive products. Points in the top-left and bottom-right contribute negative products. In this dataset, all five points fall in quadrants that yield positive products — four in top-right/bottom-left and one on the boundary (the point at the means gives zero). The covariance is the average of these signed quadrant contributions.

Pitfalls.

Covariance magnitude is uninterpretable. Never say "a covariance of 28 means a strong relationship." It means nothing about strength. Always convert to correlation for strength.
Zero covariance does not mean independence. Two variables can have a perfect non-linear relationship (a circle, a parabola) and zero covariance. Covariance only detects linear co-movement.
Outliers distort covariance heavily. A single extreme point in one quadrant can dominate the sum and flip the sign. Always plot your data before computing covariance.
Don't confuse sample and population covariance. Dividing by $n$ vs. $n - 1$ changes the value. For inference about a population, use $n - 1$ . For describing a complete dataset (the entire population), use $n$ .

Recap. Covariance measures the direction of linear co-movement between two variables. $Cov > 0$ → move together; $Cov < 0$ → move opposite; $Cov \approx 0$ → no linear pattern. The magnitude is scale-dependent and uninterpretable. Bridge: To fix the scale problem, divide covariance by the product of the two standard deviations. That gives you Pearson's correlation coefficient $r$ — a pure, unitless number between $- 1$ and $1$ .

Real-World & Domain Connection. Covariance is the building block of modern portfolio theory in finance. The covariance matrix of asset returns tells you how stocks move together — and diversification works precisely by combining assets with low or negative covariance. In ML, the covariance matrix appears in Principal Component Analysis (PCA): PCA diagonalizes the covariance matrix to find the directions of maximum variance. In deep learning, batch normalization implicitly uses covariance structure to stabilize training. Covariance is also the numerator of the slope in simple linear regression: $β_{1} = Cov (X, Y) / Var (X)$ .

12.5 Correlation Coefficient — Pearson's r

Hook. Covariance told you age and weight move together — a positive number, direction confirmed. But is the relationship tight? Loose? Covariance cannot say. You need a number that always lives between $- 1$ and $1$ , where $- 1$ means "perfectly opposite," $0$ means "no linear pattern," and $1$ means "perfectly in sync." That number is Pearson's $r$ .

Intuition + Analogy. Think of covariance as a raw score on a test — "28 points." Without knowing the maximum possible score, 28 means nothing. Correlation is like converting that raw score to a percentage. You divide by the maximum possible spread — the product of the two standard deviations. Now the number is always between $- 100%$ and $+ 100%$ . A correlation of $r = 0.9$ is like scoring 90% on the "how-linear-is-this-relationship" test.

Another analogy: covariance is like measuring the distance between two cities in "hours of travel." It depends on whether you walk, drive, or fly. Correlation is like measuring distance in "fraction of the Earth's circumference" — it is unitless and comparable everywhere.

When the analogy breaks: The percentage analogy works for interpretability but not for the math. Unlike a test score, $r$ can be exactly $- 1$ or $1$ — those are achievable with perfectly linear data. Also, $r$ is not a "percentage of relationship explained." For that, use $r^{2}$ (the coefficient of determination), which tells you the fraction of variance in Y explained by X.

12.5.1 Definition and Formula

Formalize. Pearson's $r$ is covariance normalized by the product of the two standard deviations:

$ r = \frac{Cov ( X , Y )}{σ _{X} \cdot σ _{Y}} $

Expanding both numerator and denominator:

$ r = \frac{\sum _{i = 1}^{n} ( X _{i} - X ˉ ) ( Y _{i} - Y ˉ )}{\sum _{i = 1}^{n} ( X _{i} - X ˉ ) ^{2} \cdot \sum _{i = 1}^{n} ( Y _{i} - Y ˉ ) ^{2}} $

Symbol Registry — Correlation Coefficient

$r$ — Pearson correlation coefficient — a pure number (no units), always in $[- 1, 1]$
$σ_{X}$ — sample standard deviation of $X$
$σ_{Y}$ — sample standard deviation of $Y$
$X ˉ, Y ˉ$ — sample means

Why the bounds? The Cauchy-Schwarz inequality guarantees that $∣ \sum (X_{i} - X ˉ) (Y_{i} - Y ˉ) ∣ \leq \sum (X_{i} - X ˉ)^{2} \cdot \sum (Y_{i} - Y ˉ)^{2}$ . The numerator can never exceed the denominator in absolute value. So $r \in [- 1, 1]$ is mathematically guaranteed, not just a convention.

Derivation for the age-weight example. From Section 12.4, we have:

$ Cov (X, Y) = 28, \sum (X_{i} - X ˉ)^{2} = 40, \sum (Y_{i} - Y ˉ)^{2} = ? $

Let us compute the denominator:

$ \sum (Y_{i} - Y ˉ)^{2} = (8 - 19)^{2} + (14 - 19)^{2} + (18 - 19)^{2} + (24 - 19)^{2} + (31 - 19)^{2} = (- 11)^{2} + (- 5)^{2} + (- 1)^{2} + 5^{2} + 1 2^{2} = 121 + 25 + 1 + 25 + 144 = 316 $

$ r = \frac{112}{40 \cdot 316} = \frac{112}{12640} \approx \frac{112}{112.43} \approx 0.996 $

Sense-check: $r \approx 0.996$ is extremely close to 1. The data points are five ages and weights that grow almost perfectly together. A near-perfect positive correlation is exactly what the scatterplot would show.

12.5.2 Interpretation of r Values

$r = 1$ : Perfect positive linear relationship — all points lie exactly on a line with positive slope.
$r = - 1$ : Perfect negative linear relationship — all points lie exactly on a line with negative slope.
$r = 0$ : No linear relationship (but possibly a strong non-linear one — see 12.5.5).
The sign of $r$ tells the direction (same as covariance).
The absolute value $∣ r ∣$ tells the strength.

Strength guidelines (approximate, not hard rules):

$∣ r ∣$ from 0.75 to 1.0: Strong correlation
$∣ r ∣$ from 0.5 to 0.75: Moderate correlation
$∣ r ∣$ from 0 to 0.5: Weak correlation

12.5.3 Using Correlation for Feature Selection in ML

This is where correlation becomes a direct tool in your ML workflow. Suppose you have a target $Y$ and five features $X_{1}, \dots, X_{5}$ . You compute:

$Corr (Y, X_{1}) = 0.8$ — strong
$Corr (Y, X_{2}) = 0.2$ — weak
$Corr (Y, X_{3}) = 0.6$ — moderate
$Corr (Y, X_{4}) = 0.01$ — negligible
$Corr (Y, X_{5}) = 0.1$ — weak

$X_{1}$ and $X_{3}$ are the most promising features. $X_{4}$ and $X_{5}$ look like noise — at least from a linear perspective.

The correlation heat map. You have seen this. It is an $n \times n$ grid of colored cells. Each cell $(i, j)$ shows $Corr (X_{i}, X_{j})$ . The diagonal is always 1 (a variable correlates perfectly with itself). Dark red often means strong positive, dark blue strong negative, white means near zero. Heat maps are typically the first thing you generate in exploratory data analysis — sns.heatmap(df.corr()) in Python.

Feature-vs-feature correlation (multicollinearity). If two features are strongly correlated with each other (e.g., $X_{1}$ and $X_{2}$ have $r = 0.9$ ), they are redundant. Keeping both can:

Introduce instability. The model cannot decide which feature to assign the effect to — small data changes flip the coefficients.
Inflate standard errors. The coefficient estimates become unreliable.
Add unnecessary complexity. More features without more information.

When two features are strongly correlated, consider removing one. The choice of which to remove involves domain expertise, not just the correlation number.

12.5.4 The Domain Expert Override — When Statistics Is Not Enough

A critical principle: statistical measures guide you, but they do not make the final decision. You must consult domain experts.

The brother-in-law problem. You are inviting 20 people to a party from a social circle of 1,000. Statistically, your brother-in-law might not rank in the top 20 by friendship score. But socially, you cannot exclude him. The domain expert — your spouse — overrides the statistical ranking.

The brake-pedal problem. You analyze driving data for an SUV driven mostly on highways. The brake pedal shows very low usage — low correlation with anything. Can you tell the manufacturer to remove the brakes? No. The automotive engineer knows brakes are critical for safety, even if the current dataset happens not to highlight them.

In ML terms: a feature with low correlation to the target may still be critically important for reasons the data alone cannot capture. Always validate feature-elimination decisions with the stakeholder who understands the domain.

12.5.5 Linear vs Non-Linear Relationships — A Critical Warning

When $r = 0$ , it means there is no linear relationship. It does NOT mean there is no relationship at all.

Consider data points that form a perfect circle centered at the origin. For every point $(x, y)$ , there is a point $(- x, - y)$ and also a point $(x, - y)$ — the products cancel perfectly. The correlation is exactly 0. Yet the relationship $x^{2} + y^{2} = R^{2}$ is perfectly deterministic.

Or consider a U-shaped parabola: $y = (x - 5)^{2}$ . As $x$ increases from 0 to 10, $y$ first decreases then increases. The correlation can be near zero, but the relationship is exact.

This is the biggest limitation of Pearson's $r$ : it only detects linear relationships. In the real world, most relationships are non-linear. If $r \approx 0$ , do not conclude the variables are unrelated. There may be a strong curved, periodic, or otherwise non-linear pattern.

This is also why regression — which lets you model non-linear relationships by choosing polynomial, exponential, or other functional forms — is the natural next step after correlation.

12.5.6 Student Questions and Answers — Correlation

Q: Is the correlation coefficient computed as a single value or an array?

A: For one pair of variables (X, Y), you get a single number $r$ . But when you have many features, you compute $r$ for every pair. If you have 10 variables, you compute $10 \times 10 = 100$ pairwise correlations (only 45 are unique — the matrix is symmetric, and the diagonal is always 1). That is why the correlation heat map is an $n \times n$ grid. Each cell is a single $r$ value for one pair of variables.

Q: In real-world problems, most relationships are non-linear. How do we handle that if Pearson's $r$ only measures linear relationships?

A: This is exactly the limitation of Pearson's $r$ . The answer leads into regression, where you can model non-linear relationships by choosing appropriate functional forms — polynomial ( $y = a x^{2} + b x + c$ ), exponential ( $y = a e^{b x}$ ), logarithmic, and so on. There are also rank-based correlation measures like Spearman's $ρ$ and Kendall's $τ$ that can detect monotonic (but not necessarily linear) relationships. Spearman's $ρ$ is Pearson's $r$ applied to the ranks of the data instead of the raw values — it captures any monotonic trend, not just straight lines.

Q: If $r = 0.2$ , is that weakly or moderately correlated?

A: 0.2 is generally considered a weak positive correlation. Rough guidelines: 0 to 0.5 is weak, 0.5 to 0.75 is moderate, and 0.75 to 1.0 is strong. These are not hard rules — they are heuristics. Context matters. In some fields (like psychology), $r = 0.3$ can be considered meaningful. In physics, you might expect $r > 0.99$ for a well-established law. Always interpret $r$ in the context of your domain.

Q: Does $r = 0$ mean the variables are independent?

A: No. $r = 0$ means there is no linear relationship. The variables could still be related through a non-linear pattern — a circle, a parabola, a sine wave. Independence is a stronger condition: it means knowing X tells you nothing at all about the distribution of Y. Zero correlation is necessary for independence (if variables are independent, $r = 0$ ), but it is not enough — you can have $r = 0$ with strong dependence. This is one of the most important distinctions in the course and a classic exam question.

Q: Can we get more practice problems on covariance and correlation?

A: Yes. The computational part is straightforward — find means, deviations, products, sum, and divide. The interpretation is what exam questions usually target. Focus on four key exam points. (1) What the sign of $r$ tells you about direction. (2) What the magnitude of $∣ r ∣$ tells you about strength. (3) Why $r = 0$ does not imply independence. (4) Why covariance's magnitude is uninterpretable while $r$ 's magnitude is.

Scope. Pearson's $r$ assumes:

Linear relationship. It only measures linear association. If the true relationship is curved, $r$ will underestimate the strength of association.
Interval or ratio data. Both variables should be continuous and measured on scales where differences are meaningful.
No extreme outliers. A single outlier can dramatically change $r$ . Always plot the data first.
Bivariate normality (for inference). If you want to test whether $r$ is significantly different from zero, the standard test assumes both variables are normally distributed. For large samples, this assumption relaxes.

What breaks when assumptions fail:

Non-linear relationships → $r$ is misleadingly close to zero. Use Spearman's $ρ$ or fit a non-linear model.
Outliers → $r$ can be pulled toward or away from zero dramatically. Use resistant correlation measures.
Ordinal data → Pearson's $r$ is inappropriate. Use Spearman's or Kendall's rank correlation.

Visual Intuition. Imagine a scatterplot with $X$ on the horizontal axis and $Y$ on the vertical axis. $r = 1$ : all points lie on a straight line sloping upward. $r = 0.8$ : a tight upward-pointing cigar shape. $r = 0.3$ : a diffuse upward-pointing cloud. $r = 0$ : a round, shapeless cloud — or a perfect circle, or a U-shape. $r = - 0.8$ : a tight downward-pointing cigar. The tighter the cigar, the closer $∣ r ∣$ is to 1. The rounder the cloud, the closer $∣ r ∣$ is to 0.

Pitfalls.

Correlation is not causation. $r$ close to 1 does not mean X causes Y. Ice cream sales and drowning deaths are positively correlated — because both increase in summer. A third variable (temperature) drives both.
Restricted range hides correlation. If you only look at a narrow slice of X values, $r$ can appear near zero even when the full-range relationship is strong. SAT scores and college GPA correlate about 0.5 overall — but within a single highly selective university, the correlation can be near zero because everyone has high SAT scores.
Ecological correlation (aggregation bias). Correlations computed on group averages can be much larger than correlations on individuals. The correlation between income and education at the country level is not the same as at the individual level.
Don't use $r$ for binary or categorical data. Pearson's $r$ requires continuous variables. For categorical data, use chi-square, Cramér's V, or the phi coefficient.
$r^{2}$ , not $r$ , measures "variance explained." An $r$ of 0.7 sounds impressive — but $r^{2} = 0.49$ , meaning only 49% of the variance in Y is explained by X. The other 51% is unexplained. Always square $r$ to get the proportion of shared variance.

Recap. Pearson's $r$ is covariance normalized to $[- 1, 1]$ . The sign gives direction, the absolute value gives linear strength. $r = 0$ means no linear relationship — but potentially a strong non-linear one. Correlation is the first tool in any data exploration workflow — it ranks features, exposes redundancy, and tells you where to look next. Bridge: Correlation measures association. The next step — regression — models the functional form of the relationship and lets you predict Y from X, including non-linear forms that Pearson's $r$ cannot capture.

Real-World & Domain Connection. Correlation is everywhere in data science. Finance: the correlation matrix of stock returns drives portfolio optimization. Genomics: gene co-expression networks are built from pairwise correlations. Recommender systems: user-user and item-item correlations power collaborative filtering. In ML pipelines, df.corr() is often the very first line of code after loading data — it surfaces which features might matter, which are redundant, and where to investigate further. The correlation heat map is the most common figure in any Kaggle notebook's exploratory data analysis section. But the most important lesson is the one this lecture emphasized: correlation measures linear association only. The real world is curved. Use your eyes — always plot the data.

Exam Guidance Summary

Exam note: ANOVA numerical problems are unlikely in exams due to heavy computation. The focus is on conceptual understanding: when to use ANOVA, what the F-test does, the two kinds of variance (between-group and within-group), and interpreting results. ANOVA may appear in assignments where you have time to work through the computations.

Exam note: Chi-square computations may appear in exams, especially for $2 \times 2$ or $2 \times 3$ contingency tables. Know how to compute expected frequencies, the chi-square statistic, degrees of freedom $(R - 1) (C - 1)$ , and how to compare the computed $χ^{2}$ against the critical value from the table. The worked examples in Section 12.2 are representative of what you might see.

Exam note: Covariance and correlation computational problems are very likely in exams. They are computationally straightforward — find means, deviations, products, sum, and divide. Focus your study on interpretation. Know what the sign of $r$ means for direction. Know what the magnitude means for strength. Know why "zero correlation" does not mean "no relationship." And know why covariance magnitude is uninterpretable while correlation magnitude is.

Exam note: The P-value approach may appear as a conceptual question. Know the decision rule cold: if $p \leq α$ , reject $H_{0}$ ; if $p > α$ , do not reject. Understand that the P-value is the probability of the observed result (or more extreme) assuming $H_{0}$ is true — NOT the probability that $H_{0}$ is true.

Exam note: Know the decision tree for selecting the correct test:

Means (one or two) → Z-test or t-test
Proportions (one or two) → Z-test
Several proportions / categorical association → Chi-square test
Several means (three or more) → ANOVA (F-test)

This framework is a likely exam question — either as a direct "which test would you use" question or embedded in a larger problem.

Exam note: The distinction between "no linear relationship" ( $r = 0$ ) and "no relationship at all" is a classic exam question. Be prepared to explain why $r = 0$ does not imply independence, and give a concrete example (a circle, a parabola, a sine wave). Similarly, "correlation does not imply causation" is a perennial favorite — have a clear example ready.

Exam note: Domain expertise as a complement to statistical measures is an important conceptual point. The professor's thought experiments (brother-in-law problem, brake-pedal problem) illustrate that statistical metrics alone are not enough — domain knowledge must guide feature selection and model interpretation.

Key Industry Applications

Chi-square. Testing associations in survey data and market research — does gender relate to product preference? A/B testing with categorical outcomes — do conversion rates differ across multiple landing page variants? Medical studies — does a treatment group show different recovery patterns than a control group? Quality control — are defect types independent of production shifts? In ML, sklearn.feature_selection.chi2 scores categorical features against the target for feature selection. The chi-square test of independence, invented by Karl Pearson in 1900, remains one of the most widely used statistical tools across all industries.

ANOVA. Validating regression models in ML — the overall F-test in any statsmodels regression summary is ANOVA. Comparing A/B test variants when there are more than two treatment arms. Quality control in manufacturing — comparing outputs across multiple production lines or shifts. Agricultural experiments — comparing crop yields across different treatments, fertilizers, or seed varieties (this is where R.A. Fisher invented ANOVA at Rothamsted Research in the 1920s). Pharmaceutical trials — comparing drug efficacy across multiple dosage levels. In tech, any experiment with more than two variants (A/B/C/D testing) uses ANOVA before pairwise post-hoc comparisons.

Covariance. Portfolio theory in finance — the covariance matrix of asset returns is the foundation of diversification and modern portfolio optimization. Principal Component Analysis (PCA) — PCA diagonalizes the covariance matrix to find directions of maximum variance in the data. Linear regression — the slope coefficient $β_{1} = Cov (X, Y) / Var (X)$ . Batch normalization in deep learning implicitly uses covariance structure to stabilize training. Covariance is the fundamental building block — every multivariate statistical method starts with the covariance matrix.

Correlation. Feature selection in ML — df.corr() is often the very first line of code after loading data, surfacing which features relate to the target and which are redundant. Heat-map visualizations — sns.heatmap(df.corr()) appears in most exploratory data analysis notebooks. Finance — correlations between asset returns drive portfolio risk calculations. Genomics — gene co-expression networks are built from pairwise correlations. Recommender systems — user-user and item-item correlations power collaborative filtering. Pearson's $r$ remains the most commonly reported measure of association across all scientific fields. This holds despite its limitation to linear relationships. It is simple, interpretable, and serves as the starting point for deeper modeling.

ISM Lecture 12 notes · Hypothesis Testing — ANOVA, Chi-Square, Correlation and Regression

Introduction to Statistical Methods· postgraduate· 2026-07-07

Sections Breakdown

112.1 P-value Approach to Hypothesis Testing

The P-value as an alternative to critical-region testing — computing tail probabilities and making decisions by comparing p against the significance level.

212.2 Chi-Square Test

Testing association between categorical variables using contingency tables, expected frequencies, and the chi-square statistic with worked examples.

312.3 ANOVA — Analysis of Variance

Comparing three or more group means by partitioning total variation into between-group and within-group components, using the F-test.

412.4 Covariance

Measuring the direction of linear co-movement between two variables — understanding why only the sign, not the magnitude, is interpretable.

512.5 Correlation Coefficient — Pearson's r

Normalizing covariance to a unitless number between -1 and 1 — interpreting strength, handling non-linear relationships, and using correlation for feature selection in ML.

6Exam Guidance Summary

Exam strategy and likely question patterns for hypothesis testing, chi-square, ANOVA, covariance, and correlation.

7Key Industry Applications

Real-world uses of chi-square, ANOVA, covariance, and correlation across finance, healthcare, ML, and technology.

Postgraduate students in statistics and machine learning

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

P-value Approach to Hypothesis Testing

Must-know: The P-value is the probability of observing data at least as extreme as yours, assuming H₀ is true. The decision rule: reject H₀ if p ≤ α, do not reject if p > α. The P-value is NOT the probability that H₀ is true.

$p \leq α ⟹ Reject H_{0}, p > α ⟹ Do not reject H_{0}$

⚠️ Top pitfall: Confusing the P-value with the probability that the null hypothesis is true. The P-value always assumes H₀ is true — it does not tell you whether H₀ is correct.

Self-check: If your P-value is 0.03 and α = 0.05, do you reject H₀? What if α = 0.01?

Connects to: Chi-square test, ANOVA (both report P-values); decision tree for hypothesis testing

Chi-Square Test for Independence

Must-know: The chi-square test assesses whether two categorical variables are independent. H₀: variables are independent. The test statistic is χ² = Σ (O − E)² / E with df = (R−1)(C−1). Reject H₀ if computed χ² exceeds the critical value from the chi-square table.

$χ^{2} = i = 1 \sum R j = 1 \sum C \frac{( O _{ij} - E _{ij} ) ^{2}}{E _{ij}}, E_{ij} = \frac{( e x t R o w _{i} total ) \times ( e x t C o l _{j} total )}{e x t G r an d t o t a l}$

⚠️ Top pitfall: Using chi-square when expected frequencies are too small (below 5). This breaks the approximation. Also: a significant result does not imply causation or tell you the strength of association.

Self-check: In a 3×4 contingency table, what are the degrees of freedom for the chi-square test?

Connects to: P-value approach (same decision rule), ANOVA (both compare observed vs. expected patterns), feature selection in ML

ANOVA — Analysis of Variance

Must-know: ANOVA compares three or more group means using the F-test. H₀: all means are equal. Total variation is partitioned into between-group (SSTR) and within-group (SSE) components. F = MSTR / MSE = [SSTR/(k−1)] / [SSE/(n−k)]. Reject H₀ if F exceeds the critical value from the F-table.

$F = r a c M S T R M S E = r a c j = 1 \sum k n_{j} (a r X_{j} - a r X)^{2} / (k - 1) j = 1 \sum k i = 1 \sum n_{j} (X_{ij} - a r X_{j})^{2} / (n - k)$

⚠️ Top pitfall: ANOVA only tells you that at least one mean differs — it does not identify which groups differ. Use post-hoc tests (Tukey’s HSD) for pairwise comparisons. Also: ANOVA tests means, not variances, despite the name.

Self-check: With 4 groups and 40 total observations, what are the numerator and denominator degrees of freedom for the F-test?

Connects to: Chi-square (both avoid multiple comparisons), F-test in regression summaries, two-way ANOVA extends to two factors

Covariance

Must-know: Covariance measures the direction of linear co-movement between two variables. Cov > 0 means they move together; Cov < 0 means they move opposite. Only the sign is interpretable — the magnitude is scale-dependent and meaningless on its own. Zero covariance does not imply independence.

$Cov (X, Y) = \frac{\sum _{i = 1}^{n} ( X _{i} - X ˉ ) ( Y _{i} - Y ˉ )}{n - 1}, Cov (X, X) = Var (X)$

⚠️ Top pitfall: Saying “a covariance of 28 means a strong relationship.” The magnitude is uninterpretable without context. Always convert to correlation for strength. Also: zero covariance means no linear relationship, not no relationship.

Self-check: If Cov(X,Y) = −15, what does the sign tell you? What does the magnitude not tell you?

Connects to: Pearson’s r (covariance normalized), linear regression slope β₁ = Cov(X,Y)/Var(X), PCA (covariance matrix diagonalization)

Pearson’s Correlation Coefficient (r)

Must-know: Pearson’s r is covariance normalized to [−1, 1]. The sign gives direction, the absolute value gives linear strength. r = 0 means no linear relationship — but potentially a strong non-linear one (circle, parabola). r = 0 does NOT imply independence. Correlation is not causation.

$r = \frac{e x t C o v ( X , Y )}{σ _{X} \cdot σ _{Y}} = \frac{\sum ( X _{i} - X ˉ ) ( Y _{i} - Y ˉ )}{\sum ( X _{i} - X ˉ ) ^{2} \cdot \sum ( Y _{i} - Y ˉ ) ^{2}}$

⚠️ Top pitfall: Assuming r ≈ 0 means no relationship. It only means no linear relationship. Always plot the data. Also: r², not r, measures the proportion of variance explained (r = 0.7 means r² = 0.49, so only 49% of variance is shared).

Self-check: If r = 0 for X and Y, does it mean X and Y are independent? Give a counterexample.

Connects to: Covariance (unnormalized counterpart), chi-square (alternative for categorical data), feature selection in ML, multicollinearity detection

Previous LectureLecture 11 · Hypothesis Testing

Next LectureLecture 13 · Covariance, Correlation, and Linear Regression

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Prerequisite Knowledge

Previously Covered in This Subject

Hypothesis Testing: ANOVA, Chi-Square, Correlation and Regression

12.1 P-value Approach to Hypothesis Testing

12.1.1 The Two Approaches to Decision Making

12.1.2 Symbol Registry — P-value Approach

12.1.3 When to Use Each Approach

12.2 Chi-Square Test

12.2.1 Overview and When to Use

12.2.2 Chi-Square Test for Independence

12.2.3 Chi-Square Test for Goodness of Fit

12.2.4 Formula and Computation

12.2.5 Worked Example 1 — Accident Timing and Fatality

12.2.6 Worked Example 2 — Smoking Habits and Hypertension

12.2.7 Student Questions and Answers — Chi-Square

12.3 ANOVA — Analysis of Variance

12.3.1 Overview and When to Use

12.3.2 The Two Kinds of Variance

12.3.3 The ANOVA Computation Framework

12.3.4 Worked Example — Three Groups

12.3.5 Understanding the F-Table

12.3.6 Two-Way ANOVA

12.3.7 ANOVA in the Machine Learning Context

12.3.8 Student Questions and Answers — ANOVA

12.4 Covariance

12.4.1 Definition and Formula

12.4.2 Worked Example — Age and Weight

12.4.3 Interpretation of Covariance

12.5 Correlation Coefficient — Pearson's r

12.5.1 Definition and Formula

12.5.2 Interpretation of r Values

12.5.3 Using Correlation for Feature Selection in ML

12.5.4 The Domain Expert Override — When Statistics Is Not Enough

12.5.5 Linear vs Non-Linear Relationships — A Critical Warning

12.5.6 Student Questions and Answers — Correlation

Exam Guidance Summary

Key Industry Applications

ISM Lecture 12 notes · Hypothesis Testing — ANOVA, Chi-Square, Correlation and Regression

Sections Breakdown

Exam Revision Notes

P-value Approach to Hypothesis Testing

Chi-Square Test for Independence

ANOVA — Analysis of Variance

Covariance

Pearson’s Correlation Coefficient (r)

Was this lecture useful?

BitsNotes AI Assistant

Configure AI Chat

Sign in to use AI Chat