—

Advanced Statistical Methods

Analysis of Variance and Regression

Published: 2026-08-11

Level: postgraduate

Audience: Postgraduate students in statistics and data science

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Hypothesis testing and P-values — covered in Lecture 4 (Hypothesis Testing Procedures)
The one-sample t-test — covered in Lecture 4 (Hypothesis Testing Procedures)
Z-scores as the foundation for T, chi-square, and F — covered in Lecture 3 (Statistical Inference and Hypothesis Testing)
Chi-square applications and the road to ANOVA — covered in Lecture 5 (Inferences on the Population Variance)

8.1 One-Way Analysis of Variance and the ANOVA Table

8.1.1 Why Analysis of Variance Exists

Hook: You want to compare three, four, or five production methods and decide whether they really differ in average output. Why can't you just run a t-test for every pair of methods?

Whenever we want to check for significant differences among three or more population means together, the standard tool is analysis of variance (often shortened to ANOVA). The name sounds odd — we are comparing means, so why talk about variance? The idea is that the variation inside the data carries the evidence: if the treatment means are really different, then a large part of the total variation must come from differences between the treatments rather than from chance. So we do not compare means pairwise; we decompose the total variation in the data and see which part the treatments can explain.

The alternative — running a t-test for every pair of means — becomes painful fast. With five methods, for example, there are $(2 5) = 10$ pairs to check, and each test runs its own risk of a wrong conclusion (a false alarm, a "Type I error", at the chosen significance level). Ten tests at 5% significance each do not cost 5% overall — each extra test adds another chance of error, so the total chance of at least one false conclusion climbs well above 5%. Comparing means through variation avoids that pile of pair tests: one test on the whole data set replaces ten separate ones.

Intuition and analogy: Think of each treatment as a marksman firing at the same target. Each marksman's darts are the observations; the bullseye they are aiming at is the population mean. ANOVA compares two spreads: (1) how far apart the marksmen's average dart positions are from each other (the between-treatments spread), and (2) how far individual darts scatter around each marksman's own average (the within-treatment spread). If every marksman aims at the same spot, the averages differ only by luck, so the two spreads are similar in size and their ratio is about 1. If one marksman is actually aiming elsewhere, the averages spread far apart relative to the scatter of individual darts, and the ratio grows large. A big ratio is the evidence that the aims (means) really differ.

Two procedures are used to analyze the significant differences among several population means: one-way analysis of variance, where we consider a single factor (the treatments), and two-way analysis of variance, where we consider two factors at once. Factorial designs are not part of this course — you can omit them. For the one-way discussion, the complete variation is partitioned into exactly two components: one due to the treatments (which is in our control) and one due to error, chance causes, everything not in our control.

Notation we will use throughout:

$K$ — the number of treatments (also called the number of levels of the factor). In the first example $K = 5$ .
$N_{T}$ — the total sample size, $N_{T} = N_{1} + N_{2} + \dots + N_{K}$ , where $N_{i}$ is the sample size of treatment $i$ . Here $N_{T} = 5 \times 7 = 35$ .
$μ_{1}, μ_{2}, \dots, μ_{K}$ — the population means of the treatments, the quantities we want to compare.
$x ˉ_{j}$ — the sample mean of treatment $j$ , the estimate of $μ_{j}$ from the data.

8.1.2 The ANOVA Table Layout

The results of both one-way and two-way analysis are expected to be presented in the form of an ANOVA table (analysis of variance table). Its columns are, in order:

Source of variation — which component of the total variation a row accounts for (treatments, error, total).
Sum of squares — the amount of variation in that component.
Degrees of freedom — how many independent pieces of information that component uses.
Mean square — the sum of squares divided by its degrees of freedom.
F value — the test statistic, a ratio of mean squares.
P-value — the probability that decides the conclusion.

For the sums of squares, the notation matters. $S S T$ is the total sum of squares — the total variation in the data. $S S T R$ is the sum of squares due to the treatments — the variation the treatments can explain. The remainder is $S S E$ , the sum of squares due to error. The three always satisfy the decomposition:

$S S T = S S T R + S S E$

So the total variation splits into two components: the explained part (treatments) and the unexplained part (error, left to chance). "This is always true" — the identity holds for every data set. The same decomposition is true for the degrees of freedom: the total degrees of freedom split into treatment degrees of freedom plus error degrees of freedom.

The degrees of freedom follow a fixed pattern. Treatments use $K - 1$ . The total uses $N_{T} - 1$ . The error uses the remainder, which is $N_{T} - K$ . These lead to the mean squares and the F statistic:

$M S T R = \frac{S S T R}{K - 1}, M S E = \frac{S S E}{N _{T} - K}, F = \frac{M S T R}{M S E}$

where $M S T R$ is the mean square due to treatments, $M S E$ is the mean square due to error (the variance estimate inside the samples), and $F$ is the test statistic that we compare against the F distribution.

Why F measures what it measures. The two mean squares are two independent estimates of the same quantity — the common population variance $σ^{2}$ of the response inside the treatments:

$M S T R = S S T R / (K - 1)$ measures the between-treatments variability: how far the treatment sample means $x ˉ_{j}$ spread around the overall mean. If the null hypothesis (all $μ_{j}$ equal) is true, the treatment means differ only by sampling luck, and $M S T R$ is a fair estimate of $σ^{2}$ . If some $μ_{j}$ really differ, the means spread extra wide and $M S T R$ overestimates $σ^{2}$ .
$M S E = S S E / (N_{T} - K)$ measures the within-treatment variability: how far individual observations scatter around their own treatment mean. This spread is pure chance, so $M S E$ is a fair estimate of $σ^{2}$ whether or not the null is true.

So when the null is true, the ratio $F = M S T R / M S E$ should sit near 1; when the null is false, the numerator is inflated and $F$ grows large. The F distribution provides the ruler: a large F lands in the right tail of the F distribution with $K - 1$ numerator and $N_{T} - K$ denominator degrees of freedom, and the P-value — the chance of seeing an F at least that large when all means are equal — comes out small.

There is no need for any theoretical discussion of the F distribution itself beyond this — the working method is to read the ANOVA table and comment on it.

Exam note: you may be given an incomplete ANOVA table — a layout with no specific data inputs — and asked to fill it. The simple layout above (source, sum of squares, degrees of freedom, mean square, F, P-value) is exactly what you must know how to complete. Both the one-way and the two-way classifications are examined in the form of an ANOVA table.

8.1.3 Worked Example: Filling an Incomplete ANOVA Table

This is a fundamental design: assign different treatments at random, with a single factor in view. The factor could be different methods or different machines. In the example, the factor is five methods, which you can treat as method A, method B, method C, method D, method E — five treatments. The goal is to check whether there is a significant difference among the population means of the five methods: not all means are the same.

Each method was applied to seven experimental units, so all five treatments have equal sample size $N_{1} = N_{2} = \dots = N_{5} = 7$ , and the total sample size is $N_{T} = 35$ . The textbook example gives two of the sums of squares: $S S T R = 300$ and $S S T = 460$ .

Careful with one detail: the total sum of squares is 460, not 400. The 400 that appears in the source is a typographical error; the correct textbook value is explicitly 460. If your arithmetic gives $S S T = 460$ and $S S E = 160$ , trust it — and on the exam, work from the numbers you are given.

Now fill the table step by step:

Worked example: filling the incomplete ANOVA table (five methods, seven units each).

Step 1 — Error sum of squares. From $S S T = S S T R + S S E$ :

$S S E = S S T - S S T R = 460 - 300 = 160$

Step 2 — Degrees of freedom. Treatments: $K - 1 = 5 - 1 = 4$ . Total: $N_{T} - 1 = 35 - 1 = 34$ . Error: the remainder, $34 - 4 = 30$ , which matches the formula $N_{T} - K = 35 - 5 = 30$ .

Step 3 — Mean squares. Divide each sum of squares by its degrees of freedom:

$M S T R = \frac{S S T R}{K - 1} = \frac{300}{4} = 75, M S E = \frac{S S E}{N _{T} - K} = \frac{160}{30} = 5.33$

Step 4 — F value. Divide the treatment mean square by the error mean square:

$F = \frac{M S T R}{M S E} = \frac{75}{5.33} = 14.07$

The completed ANOVA table:

Source of variation	Sum of squares	Degrees of freedom	Mean square	F
Treatments (methods)	300	4	75	14.07
Error	160	30	5.33
Total	460	34

Sense-check: the rows must add up in both of the first two columns — $300 + 160 = 460$ and $4 + 30 = 34$ — and the F value is a large number relative to 1, which already hints that the treatment means differ. Everything is consistent.

You do not always need to compute the whole table by hand — at times you will be handed an incomplete table and asked only to fill the missing entries, as we just did.

Reading the P-value. With 4 and 30 degrees of freedom, the F table shows that the area to the right of 7.84 is about 0.01 (this is the table value used in class). Our computed F = 14.07 sits far to the right of 7.84, so the area to its right is much smaller than 0.01 — the P-value is well under 0.01. The standard F tables give an even lower threshold: the area to the right of about 4.02 is 0.01 when the degrees of freedom are (4, 30), so whatever table you hold, the conclusion is the same and F = 14.07 is far beyond the 0.01 mark. With P below 0.01, the P-value is certainly less than any usual level of significance (0.05 or 0.01), so the conclusion is to reject the null hypothesis of equal means.

Visual intuition: picture the F distribution with 4 and 30 degrees of freedom — a curve that starts at zero, rises quickly to a peak near 1, then falls away in a long right tail. Draw the mark at 14.07 on the far right of that tail. The P-value is the area under the curve to the right of that mark: a sliver. Because the marksman analogy says "ratio near 1 means same aim", an F of 14.07 — fourteen times the noise level — is overwhelming evidence that at least one treatment mean differs.

The point of this exercise is not to pin down the exact P-value but to show how the whole ANOVA table gets filled — this is the one-way handling you must be able to do.

8.1.4 Equal and Unequal Sample Sizes

In the example, every treatment has exactly seven observations — this is called equal sample sizes. It need not be seven, and the sizes need not even be equal: treatment A might have four observations, treatment B seven, treatment C five, and so on. ANOVA still works with unequal sample sizes — everything simply adjusts through the degrees of freedom. You should be very careful with the degrees of freedom when the samples are unequal; that is where the adjustment shows up: the error degrees of freedom are always $N_{T} - K$ (total observations minus number of treatments), so when the sample sizes differ, only the total $N_{T}$ changes and the degrees of freedom follow along.

Scope and assumptions (when ANOVA applies and when it breaks). Analysis of variance rests on three assumptions:

Normality — the response is normally distributed within every treatment population. (With equal sample sizes, ANOVA is fairly safe even when normality fails a little.)
Equal variances — the population variance $σ^{2}$ is the same inside every treatment. This is the assumption that makes "two estimates of the same $σ^{2}$ " meaningful: if one treatment is far more variable than the others, $M S E$ is a blend and the F test loses its accuracy.
Independence — the observations are independent of one another: one worker's output does not depend on another's.

If the equal-variance or independence assumptions fail badly, the F test can reject or accept the null for the wrong reason. The sample sizes themselves do not need to be equal — that is handled by the degrees of freedom, not by the assumptions.

Visual intuition: with unequal samples, nothing about the arithmetic changes — draw the same picture of group means spread around the overall mean, but now the treatment mean $x ˉ_{j}$ with a larger sample is a more reliable estimate of its $μ_{j}$ . The formulas already weight this automatically through the degrees of freedom.

Pitfalls.

Using 400 instead of 460. The typo in the source is a classic trap; the correct total sum of squares is 460, giving $S S E = 160$ . On the exam, trust the numbers printed in the question.
Wrong degrees of freedom with unequal samples. The error degrees of freedom are $N_{T} - K$ , never $K$ and never $K (N - 1)$ unless all samples are equal. With unequal sizes, students routinely miscompute $N_{T}$ .
Mixing up $S S T R$ and $S S E$ . $S S T R$ is the between-treatment (explained) part, $S S E$ the within-treatment (unexplained) part; the F ratio is $M S T R / M S E$ , not the reverse.
Forgetting that F is a right-tail test. Only large values of F (far from 1) reject the null. A tiny F means the treatments explain less than noise, which is not evidence against the null.

Recap and bridge. One-way ANOVA decomposes the total variation into the part explained by the treatments and the part left to chance, summarized in the ANOVA table: $S S T = S S T R + S S E$ with degrees of freedom $K - 1$ , $N_{T} - K$ , $N_{T} - 1$ , mean squares $M S T R$ and $M S E$ , and the test statistic $F = M S T R / M S E$ . A large F — P-value below the level of significance — rejects the null that all treatment means are equal. That rejection only tells us that the means differ, not which pair differs: the next step, the F-distribution hypothesis test and then Fisher's LSD, answers exactly that question.

Real-world and domain connection. Manufacturing plants use this exact layout to compare production methods or machine settings: assign each method to a random set of experimental units, collect the outputs, and let the ANOVA table decide whether the methods differ in average output before any money is spent on retooling. The same machinery powers quality comparisons across suppliers, A/B tests with several variants, and agricultural field trials — anywhere the question is "do three or more groups really have different averages?" ANOVA is the first tool reached for, and the ANOVA table is how the answer is reported.

8.2 Hypothesis Testing with the F-Distribution

8.2.1 The Hypotheses

Hook: The ANOVA table ends with an F value — but what do you actually do with it? The whole table is a single hypothesis test in disguise, and the F value is its test statistic.

The one-way ANOVA answers a single question, expressed as a hypothesis test. The null hypothesis claims every treatment population mean is the same:

$H_{0} : μ_{1} = μ_{2} = \dots = μ_{K}$

and the alternative hypothesis says they are not all the same:

$H_{1} : not all means are the same$

Notice the alternative is not "all means differ from each other" — it only says at least one differs. Rejecting $H_{0}$ tells you that somewhere among the $K$ treatments, at least two means are different; it does not tell you which ones. That is exactly the division of labour in this lecture: ANOVA (the F test) establishes that a difference exists, and the multiple comparison procedure of Section 8.3 locates where.

The right-tail test. The test statistic is F from the ANOVA table, and the F distribution used here is a right-tail test: we reject the null only when F is large enough to fall in the right tail of the F distribution with $K - 1$ numerator and $N_{T} - K$ denominator degrees of freedom. The reasoning comes from Section 8.1: when $H_{0}$ is true, $M S T R$ and $M S E$ are two estimates of the same variance $σ^{2}$ , so $F = M S T R / M S E$ hovers near 1; when some $μ_{j}$ differ, $M S T R$ is inflated and F shifts right, away from the body of the distribution. Only the far right tail is evidence against the null — a tiny F near zero means the treatments explain less than the noise, which supports the null rather than attacking it.

The decision rule is the usual one: if the P-value is less than the level of significance $α$ , reject the null hypothesis. The P-value will either be given to you, or you can approximate it by reading the F distribution table — you can sense whether the P-value is greater or less than the level of significance from where the computed F falls among the table values.

8.2.2 Worked Example: The Chemtech Three-Method Problem

A smaller case: check for significant differences among the population means of three methods, method A, method B, and method C — the Chemtech example. Here $K = 3$ , and the complete ANOVA table was already built, ending at a computed F value. Because the treatments are three and the total sample size is fifteen, the F statistic carries 2 and 12 degrees of freedom: 2 is the numerator (treatments), 12 is the denominator (error), since $F = M S T R / M S E$ .

The full set-up behind the computed value: three random samples of five workers each — method A produced a sample mean of 62 units, method B 66 units, method C 52 units, with the overall mean $x ˉ = 60$ . The treatment sum of squares is

$S S T R = 5 (62 - 60)^{2} + 5 (66 - 60)^{2} + 5 (52 - 60)^{2} = 5 (4) + 5 (36) + 5 (64) = 20 + 180 + 320 = 520$

and the error sum of squares is $S S E = 340$ , giving the total $S S T = S S T R + S S E = 520 + 340 = 860$ . The mean squares are

$M S T R = \frac{S S T R}{K - 1} = \frac{520}{2} = 260, M S E = \frac{S S E}{N _{T} - K} = \frac{340}{15 - 3} = \frac{340}{12} = 28.33$

The computed value came out to:

$F = \frac{M S T R}{M S E} = \frac{260}{28.33} = 9.18 with (2, 12) degrees of freedom$

Worked example: testing the Chemtech three-method problem.

Given: $K = 3$ treatments, $N_{T} = 15$ workers, sample means 62, 66, 52, $M S T R = 260$ , $M S E = 28.33$ .

Step 1 — Hypotheses.

$H_{0} : μ_{1} = μ_{2} = μ_{3} versus H_{1} : not all means are the same$

Step 2 — Test statistic.

$F = \frac{M S T R}{M S E} = \frac{260}{28.33} = 9.18$

Step 3 — P-value from the F table. Reading the F table with 2 and 12 degrees of freedom: the area to the right of 6.93 is 0.01. Our computed F = 9.18 is larger than 6.93, so the area to its right — the P-value — is less than 0.01. (9.18 sits well beyond the 6.93 mark, so the tail probability falls below 0.01, not just close to it; software gives the exact value 0.004.)

Step 4 — Decision. Since $P < α$ for any usual $α$ (0.05 or 0.01), the rejection rule says: reject the null hypothesis. The three methods are not all the same — at least one method differs from the others in its population mean.

Sense-check: the F ratio 9.18 is far from 1, the marksman picture says "ratio near 1 means same aim", so a ratio of nine times the noise level must mean different aims. The conclusion is safe: at least one population mean differs.

Visual intuition: draw the F distribution with 2 and 12 degrees of freedom — the classic right-skewed F shape, peaking just below 1 and trailing off to the right. Mark 6.93 on the axis: the sliver of area to its right is 0.01. Now mark 9.18, further right: the sliver shrinks to less than 0.01. That shrinking sliver is the P-value, and because it has fallen below the significance level $α$ , the null hypothesis is rejected.

This conclusion sets up the next question: exactly which pair differs? That is what the multiple comparison procedure answers.

8.2.3 Reading P-Values from the F Table Without Computing Them

You do not need the exact P-value to make a decision. The procedure: look up a table value at 0.01 (or 0.05) with the right degrees of freedom; if the computed F is to the right of that table value, the P-value is smaller than 0.01 (or 0.05). This "upper tail comparison" gives you the comment you need — less than or greater than the level of significance — even when the exact probability is never computed.

For the Chemtech problem, the same comparison against the 0.05 mark works as a quick check: with (2, 12) degrees of freedom the area to the right of 3.89 is 0.05, and $F = 9.18$ is far beyond 3.89 too — so the P-value is below 0.05 as well. Whichever mark you use, the verdict is the same: reject $H_{0}$ .

Q: If we answer with P-values, are the statistical tables still necessary? A: No. The exam is open book, so every table is with you anyway. But when you give the P-value directly, you do not need to read the tables at all — compare the P-value with the level of significance and comment.

Scope. The F test assumes the ANOVA assumptions of Section 8.1 (normality, equal variances, independence) — they are what guarantee that $F = M S T R / M S E$ really follows the F distribution with $K - 1$ and $N_{T} - K$ degrees of freedom under the null. When the sample sizes are unequal, the same F test still applies; only the degrees of freedom are computed from the actual $N_{T}$ . Note also that rejecting $H_{0}$ does not say all means differ — it says at least one does, and it says nothing about which one; that question is deliberately left to Section 8.3.

Pitfalls.

Forgetting that F is one-sided. A very small F (below 1) is not evidence that the means differ; only the right tail counts. Students who "reject because F is far from 1 in any direction" get the logic backwards.
Using the wrong degrees of freedom. The numerator is $K - 1$ (treatments), the denominator is $N_{T} - K$ (error). Reading the F table with them swapped gives a different threshold.
Reading the P-value direction wrongly. If the computed F is left of the table value (e.g. 5.0 vs 6.93), the P-value is greater than 0.01 — the mark must be exceeded for the tail area to be smaller.
Claiming "all means differ" after rejection. The alternative $H_{1}$ is "not all means are the same" — at least one differs. The exact pair is found by the multiple comparison procedure, not by the F test.

Exam note: expect questions where you comment on rejecting or accepting the null hypothesis by comparing the P-value with the level of significance. State the hypotheses, give the test statistic $F$ with its degrees of freedom, compare P with $α$ , and give the verdict in words — "reject $H_{0}$ , at least one population mean differs". Because the exam is open book, tables are available; if you answer with P-values directly, the tables are not even needed.

Recap and bridge. The F test turns the ANOVA table into a decision: $F = M S T R / M S E$ against the F distribution with $(K - 1, N_{T} - K)$ degrees of freedom; a right-tail P-value below $α$ rejects equal means. The verdict "at least one differs" is only the first half of the story — the second half, finding which pair of treatments differs, is Fisher's least significant difference test in the next section.

8.3 Fisher's Least Significant Difference Test

8.3.1 When to Use LSD (and When Not To)

Hook: The F test said "at least one treatment mean differs" — but which pair? Running t-tests on every pair was rejected back in Section 8.1 as a flood of tests. Fisher's LSD does the follow-up in one disciplined pass.

Once ANOVA rejects the null hypothesis, we know some pair of treatment means differs, but not which pair. The multiple comparison procedure finds where the difference actually occurs. Many approaches exist; the one used here is the simplest, the Fisher's least significant difference test (LSD). A note on the name: the classroom label "least square difference" is sometimes used, but the standard full form is least significant difference — same test, two labels. The word "significant" is the right one, because the test checks whether an observed difference between two sample means is large enough to be statistically significant, not whether it minimizes a squared something.

The rule of thumb for when it applies: you invoke LSD only when the ANOVA conclusion rejects the null hypothesis. If the ANOVA accepts the null, there is no point in the comparative business at all — no pair needs comparing, because the data show no evidence that any means differ. So: reject first, then compare pairs. (This is why Fisher's LSD is called a protected test in the literature — the ANOVA rejection protects the pairwise stage from running on data with no differences to find.)

8.3.2 The LSD Formula and Rejection Rule

The approach compares every pair of sample means against a single threshold, the LSD value. For the pair of treatments $i$ and $j$ , with sample means $x ˉ_{i}$ and $x ˉ_{j}$ , sample sizes $n_{i}$ and $n_{j}$ , and error mean square $M S E$ , the LSD value is:

$L S D = t_{α /2} M S E (\frac{1}{n _{i}} + \frac{1}{n _{j}})$

where $t_{α /2}$ is the t-table value with the error degrees of freedom, at half the level of significance (for $α = 0.05$ , that is $t_{0.025}$ ). The error degrees of freedom are the same ones from the ANOVA table — $N_{T} - K$ — because $M S E$ is the pooled within-treatment variance estimate that carries those degrees of freedom.

Why the formula has this shape. Think of $x ˉ_{i} - x ˉ_{j}$ , the difference of two sample means. Each sample mean has standard error $M S E / n$ , so the standard error of their difference is $M S E / n_{i} + M S E / n_{j} = M S E (1/ n_{i} + 1/ n_{j})$ — the two sampling errors add up in squares, like the legs of a right triangle. Multiply that standard error by the t critical value $t_{α /2}$ and you get the largest difference that could plausibly be pure sampling luck at significance $α$ . Any difference beyond that threshold is declared real. Notice the ingredients: $M S E$ carries the within-treatment noise, $n_{i}$ and $n_{j}$ carry the sample sizes, and $t_{α /2}$ carries the confidence level.

The rejection rule for a pair:

$Reject H_{0} for pair (i, j) if ∣ x ˉ_{i} - x ˉ_{j} ∣ \geq L S D$

The absolute value matters: we consider only the magnitude of the difference, ignoring its sign. If $x ˉ_{1} = 2$ and $x ˉ_{2} = 1$ , the modulus $∣2 - 1∣$ gives 1; the other way around, $x ˉ_{1} = 1$ and $x ˉ_{2} = 2$ , $∣1 - 2∣$ also gives 1. Either direction, the comparison uses 1. So even if a difference turns out negative, you take its positive value and compare that with the LSD.

The textbook also offers a t-based variant of the same test, which computes a t statistic for each pair directly:

$t = \frac{x ˉ _{i} - x ˉ _{j}}{M S E ( \frac{1}{n _{i}} + \frac{1}{n _{j}} )}$

with the same rejection logic as a two-tailed t test on $N_{T} - K$ degrees of freedom. It is mathematically the same comparison: $∣ t ∣ \geq t_{α /2}$ holds exactly when $∣ x ˉ_{i} - x ˉ_{j} ∣ \geq L S D$ . The simple difference approach above is preferred here because it lets you conclude where the difference occurs in one line — one LSD number for the whole problem, and every pair is compared against it.

A cost of doing many pairwise comparisons at once is worth knowing: each pair is tested at the comparisonwise level $α = 0.05$ , but the chance of at least one false "significant" result somewhere among all pairs (the experimentwise error rate) grows with the number of pairs. With three treatments there are $(2 3) = 3$ pairs and the overall error rate is about $1 - (0.95)^{3} \approx 0.143$ ; with five treatments, ten pairs push it to $1 - (0.95)^{10} \approx 0.40$ . This lecture's exam uses the simple difference approach as-is; the Bonferroni-style corrections that tame the experimentwise rate belong to later coursework.

8.3.3 Worked Example: Method A versus Method C

Take the Chemtech data (three methods, complete ANOVA already done). We test whether a significant difference exists between method A and method C.

Given values:

$x ˉ_{1} = 62$ (method A), $x ˉ_{3} = 52$ (method C). The difference: $∣ x ˉ_{1} - x ˉ_{3} ∣ = ∣62 - 52∣ = 10$ .
Level of significance 0.05 (if nothing else is given, use 0.05).
Error degrees of freedom 12, so the t table gives $t_{0.025, 12} = 2.179$ .
$M S E = 28.33$ from the ANOVA table.
Sample sizes: $n_{1} = n_{3} = 5$ (five observations per method; total 15, matching the 12 error degrees of freedom $N_{T} - K = 15 - 3 = 12$ ).

Worked example: is there a significant difference between method A and method C?

Step 1 — Compute the LSD value.

$L S D = t_{α /2} M S E (\frac{1}{n _{i}} + \frac{1}{n _{j}}) = 2.179 \times 28.33 (\frac{1}{5} + \frac{1}{5})$

$= 2.179 \times 28.33 \times 0.4 = 2.179 \times 11.332 = 2.179 \times 3.366 \approx 7.34$

Step 2 — Compare the sample means. The difference between the sample means is:

$∣ x ˉ_{1} - x ˉ_{3} ∣ = ∣62 - 52∣ = 10$

Step 3 — Decision. Since $10 > L S D = 7.34$ , the difference is larger than anything sampling luck could plausibly produce at the 0.05 level: there is a significant difference between method A and method C — for the pair (1, 3) we reject the null hypothesis.

Sense-check: the difference of 10 units is nearly one and a half times the threshold 7.34, so the verdict is comfortable, not borderline. If the means had been 66 and 52 (a difference of 14), the verdict would be even stronger.

8.3.4 Completing the Pairwise Comparisons

The same comparison runs for every pair, each time taking the difference of the two sample means and comparing it with the same LSD = 7.34:

Method A vs method C: difference $10 > 7.34$ → significant difference exists between population 1 and population 3.
Method B vs method C: difference $∣66 - 52∣ = 14 > 7.34$ → again a significant difference exists between population 2 and population 3.
Method A vs method B: difference $∣62 - 66∣ = 4 < 7.34$ → no significant difference between population 1 and population 2.

Pair	$∣ x ˉ_{i} - x ˉ_{j} ∣$	LSD = 7.34	Verdict
A vs C	10	10 > 7.34	Significant difference
B vs C	14	14 > 7.34	Significant difference
A vs B	4	4 < 7.34	No significant difference

The whole set of observations sums up in one sentence: methods A and B both differ from method C, but methods A and B do not differ from each other. The conclusion preserves the ANOVA rejection, then narrows it to the exact pair that drives it — method C is the odd one out.

Pitfalls.

Running LSD before the ANOVA rejects. If the F test accepts the null, stop. Comparing pairs on data that show no overall difference can manufacture "significant" pairs by luck — the comparison is only protected when it follows a rejection.
Forgetting the absolute value. The rule is $∣ x ˉ_{i} - x ˉ_{j} ∣ \geq L S D$ . A negative difference (e.g. $62 - 66 = - 4$ ) is still a magnitude of 4 and must be compared as 4.
Comparing with the wrong threshold. The LSD value is a single number computed once from $M S E$ , $t_{α /2}$ , and the sample sizes — not the F value, not the individual t values, and not recomputed differently for each pair when the sample sizes are equal.
Using $α$ instead of $α /2$ in the t table. For a 0.05-level two-tailed comparison the t value is $t_{0.025}$ , which is larger than $t_{0.05}$ — using $t_{0.05}$ makes the threshold too small and the test too eager to declare differences.

Exam note: whenever ANOVA rejects the null and you are asked whether a significant difference exists between a particular pair of treatments, use this simple difference approach — compare the absolute difference of the two sample means with the LSD value:

$L S D = t_{α /2} M S E (\frac{1}{n _{i}} + \frac{1}{n _{j}})$

This is the only multiple comparison procedure required. State the pair, state the difference, state the LSD, and give the verdict.

Recap and bridge. LSD answers the "which pair?" question left open by the F test: after an ANOVA rejection, one threshold $L S D = t_{α /2} M S E (1/ n_{i} + 1/ n_{j})$ separates real differences from sampling luck; a pair's means differ when $∣ x ˉ_{i} - x ˉ_{j} ∣ \geq L S D$ . In the Chemtech data that narrows the verdict to "A and B differ from C". The whole analysis so far has used a single factor — the next section asks what changes when a second factor (drivers, say) is added, which is the two-way ANOVA.

8.4 Two-Way Analysis of Variance

8.4.1 The VARTA Problem Setup

Hook: The one-way model said the routes explain only 31% of the travel-time variation — and left a huge pile unexplained. What if the "error" is not chance at all, but a second factor the analysis has been ignoring?

The example comes from a transit authority setting: the Varan Area Regional Transit Authority (VARTA) is expanding bus service from the suburb of Starbrick into the central business district of Varan (the route corridor is referred to as Downvaran). Four routes are being considered — the routes named are US West End, Hickory Street, and Route 59, with a fourth route in the set. VARTA ran several tests to determine whether there was a difference in the mean travel time across routes.

The response variable is mean travel time. One obvious factor is the route. But a second factor sits in the data too: the drivers. The test was set up so that each driver drove along each of the four routes. That structure matters, because a driver's driving style also moves travel time up and down, and the analysis can either ignore that influence or account for it. If every driver tested every route, then any difference between drivers — fast ones, slow ones, cautious ones — shows up in the data as a pattern the analysis can name, instead of being silently dumped into the error term.

8.4.2 One-Way Treatment: Routes Only

If we ignore the drivers and treat only the routes as the factor, a one-way ANOVA suffices: the total variation in mean travel time is split into variation explained by the different routes plus chance:

$S S T = S S T R_{routes} + S S E$

Look at the numbers from the one-way output: the total variation is $S S T = 229.2$ , and the routes account for only $72.8$ of it.

Worked example: the one-way reading of the VARTA travel-time data.

Given: the one-way ANOVA output reports total variation $S S T = 229.2$ and route (treatment) variation $S S T R_{routes} = 72.8$ .

Step 1 — Express the routes' share of the total as a percentage:

$\frac{72.8}{229.2} \approx 0.318 \approx 31%$

Step 2 — Read the leftover. The error sum of squares is everything the routes did not explain:

$S S E = S S T - S S T R_{routes} = 229.2 - 72.8 = 156.4$

so about $100% - 31% = 69%$ of the total variation sits in the error term.

Step 3 — Diagnose. Only about 31% of the total variation is explained by the different routes; the remaining huge block of variation is dumped into the error term — a massive amount of unexplained variation.

Sense-check: a 69% error share is not "mostly noise" — it is a signal. Travel times do not swing that wildly by chance; the unexplained mass points to a missing factor, and the drivers are the obvious candidate.

So only about 31% of the total variation is explained by the different routes. The remaining huge block of variation is dumped into the error term — a massive amount of unexplained variation.

Intuition: that unexplained 69% is a message, not a failure. The one-way model says "routes explain a third of the variation; the rest is luck" — but travel times that swing wildly from run to run are not luck. They carry the signature of a second influence, and in this study the obvious suspect is the driver behind the wheel. A huge error term is the standard diagnostic signal in industry: a factor is missing from the model. At the start of an analysis you might ignore a few factors; later you think again and accommodate them.

That alone is a signal: the one-way model is leaving real structure unexplained, and you should look for other factors that influence travel time. At the start of an analysis you might ignore a few factors; later you think again and accommodate them. In this problem, the drivers make good sense as an influence on travel time, and the numbers reflect that.

8.4.3 Two-Way Decomposition: Routes, Drivers, and Error

Now run a two-way ANOVA with both factors — routes and drivers. The total variation decomposes into three components instead of two:

$S S T = S S_{routes} + S S_{drivers} + S S E$

One component is due to the routes, one due to the drivers, and the remainder is the error.

The two-way structure. With $k$ treatments (routes) and $b$ blocks (drivers), and every driver driving every route, the data form a rectangular layout and the sums of squares partition cleanly:

$S S_{routes}$ — how far the route means spread around the overall mean travel time;
$S S_{drivers}$ — how far the driver means spread around the overall mean travel time;
$S S E$ — whatever is left after both factors are accounted for.

The degrees of freedom partition the same way: $k - 1$ for routes, $b - 1$ for drivers, $(k - 1) (b - 1)$ for error, and $k b - 1 = N_{T} - 1$ in total. The test for route differences uses $F = M S T R / M S E$ , the same F machinery as the one-way case.

The striking consequence: the error variation $S S E$ shrinks compared with the one-way analysis, because variation that the one-way treatment could not explain is now accounted for by the drivers factor.

A very important point: as you add factors that genuinely influence the response, the error variation decreases — SSE is smaller in the two-way analysis than in the one-way analysis. Adding a meaningful factor pulls variation out of the error pile and labels it "explained". If you go on adding the number of factors that influence the response, the error variation keeps coming down. This is the engine behind every comparison of one-way versus two-way results: the two-way ANOVA splits the same total $S S T$ into more named parts, so the leftover error is smaller and the F test for the factor you care about becomes sharper. (The gain is real only when the added factor truly matters — a meaningless factor leaves SSE almost unchanged.)

Scope of the two-way layout used here. This course treats the two-way classification with one observation per route–driver combination, the design the textbook calls the randomized block design. Because there is a single observation per cell, the analysis separates the two factors but does not estimate an interaction between them (whether some drivers are disproportionately slower on some routes); factorial designs with interactions and replications are outside the syllabus. What is examinable is the decomposition $S S T = S S_{routes} + S S_{drivers} + S S E$ , the degrees of freedom, and the one-way-versus-two-way comparison.

Visual intuition: picture the one-way result as a single column of travel-time points around the overall mean, with a thick band of scatter labeled "error". The two-way picture spreads the same points into a table — rows are drivers, columns are routes. Once the row means are drawn in, much of that thick band disappears: the scatter that looked like noise around the grand mean is actually the systematic shift of the driver rows. The error band left over is visibly thinner, and that thinning is precisely the point of the two-way analysis.

Real-world and domain connection: this is the standard reading in transit and logistics studies — a travel-time comparison across routes (or routes and shifts, routes and vehicle types) that ignores the driver factor will always overstate the error and understate the differences between routes. The same two-way machinery appears in agriculture (field blocks by soil quality), medicine (patients as blocks receiving each treatment), and marketing (stores as blocks testing displays).

Pitfalls.

Comparing one-way and two-way numbers wrongly. The total $S S T = 229.2$ is the same in both analyses — it depends only on the data, not on the model. What changes is the split: the two-way run reassigns part of the old error term to the drivers, shrinking SSE.
Reading 31% as "routes are unimportant". The 31% figure is about the one-way model's explanatory power, not about routes. Once drivers are included, the tests are rerun on the sharper two-way error, and route differences are judged against a thinner noise floor.
Forgetting the direction of the key comparison. The exam line to remember: error variation is less in the two-way analysis of variance — not more, and not equal.
Treating drivers as the factor of interest. The study was designed to test routes; the drivers were included to soak up nuisance variation and sharpen that test. The lecture's conclusion about "other factors such as drivers influencing travel time" is a diagnostic reading, not the formal test.

Q: Can we use Excel or any other software to solve the problems instead of doing everything by hand? A: Yes. The division team has confirmed that students may use Excel or any software for the problems. Some problems will still need manual work, and a mix of both is fine.

Exam note: expect a comparison question between one-way and two-way — the key line to remember is that error variation is less in the two-way analysis of variance. Both the one-way and two-way classifications are to be presented in the form of an ANOVA table, with the sums of squares, degrees of freedom, mean squares, F values, and P-values laid out in the standard columns.

Recap and bridge. Two-way ANOVA splits the same total variation into three named parts — routes, drivers, and error — instead of two. Adding the drivers factor pulls real structure out of the error pile, so SSE shrinks and the route comparison sharpens; in the VARTA data, routes explained only 31% of the variation in the one-way run, and the enormous leftover error was the clue that the drivers were the missing factor. The lecture now turns from comparing group means to a different job: predicting a quantitative outcome from another quantitative variable — simple linear regression, where the same decomposition of total variation into explained and unexplained parts reappears as $S S T = S S R + S S E$ .

8.5 Simple Linear Regression and the Decomposition of Variation

8.5.1 Recap: Best-Fit Lines and Residuals

Hook: ANOVA asked whether group means differ. Now the question changes: can one quantitative variable — population size — predict another — quarterly sales — and how much of the variation is explained by that prediction?

Simple linear regression establishes the relationship between a single independent variable and a single dependent variable, both quantitative. The relationship is estimated using the principle of least squares. Here is the motivating picture: scatter the points of $y$ against $x$ . Infinitely many lines pass through the cloud of points; we must choose one. The best fitted line is the one where the predictions are closest to the observations.

If $y^_{i}$ is the predicted value for observation $i$ and $y_{i}$ is the observed value, then the gap

$y_{i} - y^_{i}$

is the residual — the error of the prediction. If every observation lay exactly on the predicted line, every residual would be zero, there would be no error at all, and the fitted line would be perfect. In practice that does not happen, because an element of chance always exists. The main goal of regression is to find the best model — the one that keeps the error as small as possible.

Formalize: the least squares line. The population model behind the data is

$y = β_{0} + β_{1} x + ε$

where $β_{0}$ is the intercept, $β_{1}$ is the slope, and $ε$ (epsilon) is the random error term. The sample estimates $b_{0}$ and $b_{1}$ build the estimated regression equation:

$y^= b_{0} + b_{1} x$

The least squares method picks $b_{0}$ and $b_{1}$ to minimize the sum of squared residuals:

$min i = 1 \sum N (y_{i} - y^_{i})^{2} = min i = 1 \sum N (y_{i} - b_{0} - b_{1} x_{i})^{2}$

The solution is found by solving the two normal equations; the resulting formulas are:

$b_{1} = \frac{\sum _{i = 1}^{N} ( x _{i} - x ˉ ) ( y _{i} - y ˉ )}{\sum _{i = 1}^{N} ( x _{i} - x ˉ ) ^{2}}, b_{0} = y ˉ - b_{1} x ˉ$

where $x ˉ$ is the mean of the independent variable and $y ˉ$ the mean of the dependent variable. Every symbol earns its place: the numerator of $b_{1}$ measures how strongly $x$ and $y$ move together (each point's distance from the x-mean times its distance from the y-mean), and the denominator normalizes by how spread out $x$ itself is — a slope is a "change in y per change in x", so the x-spread is the unit of measurement.

This mirrors the ANOVA discussion: with one factor and a simple linear regression, you see some amount of error; if another factor is brought into the model, the error shrinks. That is exactly the point of moving from one independent variable to several — multiple linear regression (multiple explanatory variables) also helps reduce the error.

8.5.2 The Quarterly Sales Example

The running example predicts quarterly sales from the size of the population in an area. The estimated regression equation is:

$y^= 60 + 5 x$

where $x$ is the population size and $y^$ is the predicted quarterly sales. This equation came from a data set of ten observations.

Worked example: building the quarterly sales regression (ten observations).

Step 1 — The means. The ten population sizes add to $\sum x_{i} = 140$ , and the ten quarterly sales figures add to $\sum y_{i} = 1300$ , so:

$x ˉ = \frac{140}{10} = 14, y ˉ = \frac{1300}{10} = 130$

Step 2 — The slope. For each of the ten restaurants compute the product $(x_{i} - x ˉ) (y_{i} - y ˉ)$ ; they sum to 2840. The squared deviations $(x_{i} - x ˉ)^{2}$ sum to 568. So:

$b_{1} = \frac{2840}{568} = 5$

Step 3 — The intercept.

$b_{0} = y ˉ - b_{1} x ˉ = 130 - 5 (14) = 130 - 70 = 60$

Step 4 — The estimated regression equation.

$y^= 60 + 5 x$

Sense-check: a campus with 14,000 students sits at the data mean, and the line predicts $60 + 5 (14) = 130$ , exactly the mean sales — the regression line always passes through the point $(x ˉ, y ˉ)$ . A campus with 16,000 students predicts $60 + 5 (16) = 140$ thousand dollars of quarterly sales.

Now ask a different question: how much variation can we commit if we predict quarterly sales without knowing the independent variable at all? Forget the line — the line only exists when $x$ is known. Without $x$ , the best estimate of sales is simply the sample mean of the observed sales:

$y ˉ = \frac{\sum _{i = 1}^{N} y _{i}}{N} = \frac{1300}{10} = 130$

So the reference point without any independent variable is $y ˉ = 130$ , the sample mean of the ten quarterly sales observations.

Visual intuition: the scatter diagram plots quarterly sales (vertical axis, thousands of dollars) against student population (horizontal axis, thousands). The ten points climb from lower left (restaurant 1: $x = 2, y = 58$ ) to upper right (restaurant 10: $x = 26, y = 202$ ). Two horizontal-and-slanted lines appear in the picture: the flat line $y = y ˉ = 130$ and the fitted line $y^= 60 + 5 x$ . The points hug the slanted line far more closely than the flat one — the vertical gaps from the slanted line are the residuals, and the picture makes visible why the slanted line is the better predictor.

8.5.3 SST, SSR, and SSE: Three Sources of Variation

With the reference point in hand, each observation now generates two deviations:

$y_{i} - y ˉ$ — the deviation of the observed value from the sample mean, used when no independent variable is known.
$y_{i} - y^_{i}$ — the residual, the deviation of the observed value from the prediction of the regression line.

Squaring and summing each deviation gives two sums of squares. The total sum of squares is

$S S T = i = 1 \sum N (y_{i} - y ˉ)^{2}$

and the error (residual) sum of squares is

$S S E = i = 1 \sum N (y_{i} - y^_{i})^{2}$

There is a third variation accounting for the gap between the predicted values and the mean: the regression sum of squares

$S S R = i = 1 \sum N (y^_{i} - y ˉ)^{2}$

The three variations fit together exactly:

$S S T = S S R + S S E$

Why the identity holds for every data set. At the level of a single observation, the deviation from the mean splits into two pieces that add up exactly:

$y_{i} - y ˉ = (y^_{i} - y ˉ) + (y_{i} - y^_{i})$

The first piece, $y^_{i} - y ˉ$ , is how far the model's prediction sits from the flat baseline — the part of the deviation the regression line explains. The second piece, $y_{i} - y^_{i}$ , is the residual — the part the line misses. With least squares, the cross-terms that arise when you square and sum vanish, so the squared deviations add up exactly too:

$i = 1 \sum N (y_{i} - y ˉ)^{2} = i = 1 \sum N (y^_{i} - y ˉ)^{2} + i = 1 \sum N (y_{i} - y^_{i})^{2}$

That is, $S S T = S S R + S S E$ . The total variation in the dependent variable decomposes into the explained variation $S S R$ (explained by the regression, the part the model accounts for) and the unexplained variation $S S E$ (the residual error). In the quarterly sales example the total is $S S T = 15730$ ; the regression explains $S S R = 14200$ of it and leaves $S S E = 1530$ — a small leftover relative to the total, which already foreshadows a good fit.

So the decomposition works at the level of each single point and then at the level of the whole data set. These three sums of squares are exactly the ones printed in the ANOVA table of the regression output — and they are the raw material for the coefficient of determination next.

Scope and assumptions. The least squares line is the best linear description of the relationship, and its usefulness rests on assumptions about the error term $ε$ : it has mean zero for every $x$ (no systematic miss), constant variance $σ^{2}$ across all $x$ (no fanning out), independence from one observation to the next, and normality. When these fail, predictions can be biased or the fitted line can be dragged around by outliers. Also, the line should only be trusted for prediction inside the range of $x$ values actually observed — outside that range the same relationship is not guaranteed.

Pitfalls.

Confusing the two deviations. $y_{i} - y ˉ$ is the total deviation (used when no $x$ is known); $y_{i} - y^_{i}$ is the residual (used with the line). Mixing them up destroys the identity $S S T = S S R + S S E$ .
Forgetting the baseline. Without $x$ , the best prediction is $y ˉ = 130$ , not zero and not the intercept. The whole decomposition is measured against that baseline.
Thinking the identity is approximate. $S S T = S S R + S S E$ holds exactly for every data set when the line is fitted by least squares — the example's arithmetic $14200 + 1530 = 15730$ is exact, not rounded.
Reading the slope as a prediction outside the data range. A slope of 5 means sales rise by 5 thousand dollars per 1,000 students within the observed range; extrapolating far beyond it is unsafe.

Recap and bridge. Simple linear regression draws one line through the scatter of points, chosen by least squares so the squared residuals are as small as possible. Once the line is drawn, the total variation $S S T$ splits exactly into the explained part $S S R$ and the unexplained part $S S E$ . The very next question is how to summarize that split in a single number — the coefficient of determination $R^{2} = S S R / S S T$ .

8.6 Coefficient of Determination

8.6.1 Definition

The coefficient of determination — written $R^{2}$ (the output also calls it small r-square or capital R-square) — is a statistical measure in the regression model that determines the proportion of variance in the dependent variable that can be explained by the independent variable. Another way to say it: the amount of total variation explained by the estimated regression equation. It is the share of the dependent variable's variation that the given independent variable accounts for.

It is defined as the ratio of the regression sum of squares to the total sum of squares:

$R^{2} = \frac{S S R}{S S T}$

Reading the ratio. Recall from Section 8.5 that the total variation $S S T$ splits exactly into the explained part $S S R$ and the unexplained part $S S E$ . The coefficient of determination is simply the explained share of the total:

$R^{2} = \frac{S S R}{S S T} = \frac{S S T - S S E}{S S T} = 1 - \frac{S S E}{S S T}$

All three forms are the same number. The last form shows the logic in words: $R^{2}$ is 1 minus the leftover error's share. If the residuals are tiny, $S S E$ is a small slice of $S S T$ , and $R^{2}$ is close to 1; if the line explains nothing, the residuals eat the whole total and $R^{2}$ collapses toward 0. Because it is a proportion, $R^{2}$ is multiplied by 100 when spoken of as a percentage: $R^{2} = 0.9027$ means 90.27%.

8.6.2 What Happens When There Is No Error

Work through the two extreme situations to see the range of $R^{2}$ .

Perfect fit. Suppose every observed value equals its predicted value. What happens to SSE?

Q: What happens to the error if every observed value equals its predicted value — that is, if $y_{i} = y^_{i}$ ? A: The error becomes zero: each residual $y_{i} - y^_{i}$ is zero, so SSE is zero and SST equals SSR. Then $R^{2} = SSR / SST = 1$ , the largest possible value of the coefficient of determination.

So with a perfect fit, $R^{2} = 1$ — the maximum. The higher the $R^{2}$ , the better the fit; a value close to 1 means the model is doing well.

No explanatory power. The poorer the prediction, the bigger the residuals, and the smaller $S S R$ becomes. In the extreme, if the regression explains nothing, $S S R$ is near zero, and

$R^{2} = \frac{0}{S S T} = 0$

The coefficient of determination always lies between 0 and 1:

$0 \leq R^{2} \leq 1$

A value close to 1 signals the best fitted line; a value close to 0 means the model is not the best fit. Compare with the correlation coefficient, which ranges between $- 1$ and $+ 1$ ; $R^{2}$ is restricted to the non-negative range 0 to 1 because it is a proportion of variation, not a directional measure.

Visual intuition: picture the two lines from Section 8.5 — the flat baseline $y = y ˉ$ and the fitted regression line. $R^{2}$ is the fraction of the total squared gap from the flat line that the slanted line closes. At one extreme the slanted line hugs every point: it closes all the gaps, $S S E = 0$ , $R^{2} = 1$ . At the other extreme the slanted line is no better than the flat one: it closes nothing, $S S R \approx 0$ , $R^{2} \approx 0$ .

8.6.3 Worked Example: R² = 0.9027 for Quarterly Sales

The regression output (the ANOVA table of the quarterly sales regression) gives the three sums of squares. The total sum of squares is $S S T = 15730$ , and the regression sum of squares is $S S R = 14200$ — obtained exactly from the identity $S S R = S S T - S S E = 15730 - 1530 = 14200$ , since the residual sum of squares is $S S E = 1530$ .

Worked example: computing the coefficient of determination.

Given: $S S R = 14200$ , $S S T = 15730$ from the regression output of the quarterly sales data.

Step 1 — Substitute the two sums of squares into the definition:

$R^{2} = \frac{S S R}{S S T} = \frac{14200}{15730} = 0.9027$

Step 2 — Convert to a percentage.

$0.9027 \times 100 = 90.27%$

Step 3 — Interpret. Roughly 90% of the variability in quarterly sales is explained by the linear regression with population size.

Sense-check: $R^{2} = 0.9027$ is close to 1, and the picture supports it — the points cluster tightly around the fitted line $y^= 60 + 5 x$ , with small residuals and $S S E = 1530$ against a total of 15730. A near-90% explained share means the model is doing well.

Roughly 90% — that is the value printed in the regression output. The logic behind the screen: all possible variations were taken into account and the output matrix reports $R^{2} = 0.9027$ .

8.6.4 Interpreting R² in Words

In the language of the sales example: 90.27% of the variability in sales can be explained by the linear regression with the equation $y^= 60 + 5 x$ . Equivalently, about 90% of the quarterly sales variation is explained by the population size. Both phrasings answer a coefficient-of-determination question correctly — the percentage of variation in the dependent variable explained by the independent variable, or by the estimated regression equation.

Scope. The coefficient of determination describes how well the estimated regression equation fits the data actually observed. A high $R^{2}$ does not by itself prove the relationship is real, that the model is the right one, or that adding more variables improves it: you cannot conclude by looking at the R-square value alone — more tests for significance follow, and they are what validates the model. $R^{2}$ also says nothing about cause and effect: population size and sales moving together does not by itself prove one causes the other.

Pitfalls.

Reporting the ratio without the percentage. $R^{2} = 0.9027$ and "90.27%" are the same quantity, and exam answers should give the percentage interpretation in words, not just the decimal.
Treating $R^{2} = 0$ as a failure of the concept. Zero just means the line explains none of the variation — exactly the other endpoint of the 0 to 1 range.
Comparing $R^{2}$ with the correlation coefficient. The correlation coefficient lives in $[- 1, 1]$ and carries a sign; $R^{2}$ is its square-like proportion in $[0, 1]$ . The two are related — $r^{2}$ equals $R^{2}$ in simple linear regression — but the ranges differ.
Using the wrong pair of sums of squares. The ratio is $S S R / S S T$ , never $S S E / S S T$ (which would be the unexplained share) and never $S S R / S S E$ .

Exam note: for a definition-type question, write that $R^{2}$ measures the percentage of variation in the dependent variable explained by the independent variable; for a numerical question, report $R^{2}$ itself plus the percentage interpretation in words (e.g. "90.27% of the variability in sales is explained by the linear regression"). Also be ready for the perfect-fit question: when every observed value equals its predicted value, $S S E = 0$ , $S S T = S S R$ , and $R^{2} = 1$ .

Recap and bridge. The coefficient of determination $R^{2} = S S R / S S T$ condenses the whole decomposition of Section 8.5 into one number between 0 and 1: the fraction of the dependent variable's variation the regression explains. For quarterly sales that fraction is 0.9027, about 90%. The lecture has already warned that a high $R^{2}$ alone is not a verdict — the next step is the multiple regression setting, where adding variables raises $R^{2}$ automatically and the adjusted $R^{2}$ and multicollinearity enter the picture.

8.7 Multiple Linear Regression, Adjusted R², and Multicollinearity

8.7.1 Adding Variables Raises R²

A natural question follows the perfect-fit discussion:

Q: Is R-square always 1 when the fit is perfect? And does R-square keep going up as we add more independent variables? A: R-square equals 1 exactly when the fit is perfect. As the number of independent variables increases, R-square increases too — but then multicollinearity enters the picture, and that problem is addressed by the adjusted R-square.

So $R^{2}$ rises (or at least never falls) as independent variables are added: each extra variable gets credit for explaining a bit more variation, because $S S E$ can only shrink when the model has more to work with. The mechanics are simple — in the identity $S S R = S S T - S S E$ , the total $S S T$ does not change when you add a variable, while the residuals $S S E$ can only stay the same or get smaller as the model gains flexibility; a smaller $S S E$ means a larger $S S R$ , and the ratio $R^{2} = S S R / S S T$ moves up.

The catch. A bigger $R^{2}$ with more variables is not automatically a better model. A variable can raise $R^{2}$ by explaining variation that was already explained by its sibling variable — or even by fitting noise. The plain $R^{2}$ has no way to punish a model for adding useless predictors, which is why it is the wrong yardstick once several independent variables are in play. That is where the adjusted $R^{2}$ comes in; the concept was deliberately saved for the multiple linear regression session.

8.7.2 Multicollinearity and the Adjusted R²

The problem behind the scenes is multicollinearity: situations where two or more independent variables are related to each other. If two independent variables measure essentially the same thing, there is no point keeping both in the study — you can eliminate one, because the second adds no fresh information. The plain $R^{2}$ cannot account for this. The adjusted R² metric handles exactly that situation, so when many independent variables are in the picture, adjusted $R^{2}$ is the valid metric to address the comparison — you cannot conclude just by looking at the $R^{2}$ value alone.

Formalize: the adjusted coefficient of determination. With $n$ observations and $p$ independent variables, the adjusted $R^{2}$ is

$R_{a}^{2} = 1 - (1 - R^{2}) \frac{n - 1}{n - p - 1}$

where $R^{2}$ is the ordinary coefficient of determination. Compare the two formulas:

$R^{2} = 1 - S S E / S S T$ rewards every added variable, since $S S E$ never grows.
$R_{a}^{2}$ multiplies the unexplained share $(1 - R^{2})$ by the factor $(n - 1) / (n - p - 1)$ , which gets larger as $p$ grows. The penalty is small when $n$ is large and $p$ small, but it bites when many variables are fitted on few observations.

The practical reading: adding a genuinely useful variable raises $R^{2}$ enough to overcome the penalty, so $R_{a}^{2}$ rises; adding a useless variable raises $R^{2}$ only slightly, so $R_{a}^{2}$ can fall. In extreme cases — small $R^{2}$ with many variables — the formula can even go negative, and software then reports it as zero.

Multicollinearity in practice. The word "independent variable" does not mean the variables are statistically independent of each other — most predictors are correlated to some degree. The difficulty is how much. A standard rule of thumb: multicollinearity is a potential problem when the absolute value of the sample correlation coefficient between any two independent variables exceeds 0.7. When two predictors are that closely related, they carry the same information, the second one adds little, and keeping both muddies the interpretation: the model cannot cleanly separate which variable is doing the explaining. The remedy in this course's setting is the simple one — drop one of the redundant pair.

For a multiple linear regression with many independent variables, adjusted $R^{2}$ is the metric to report; the example demonstrating its influence comes in the multiple regression session.

8.7.3 Worked Example: Butler Tracking Data

The Butler tracking example compares two regressions on the same data:

Model 1, one independent variable: $y^= b_{0} + b_{1} x_{1}$ , where $b_{0}$ and $b_{1}$ are the estimated coefficients.
Model 2, two independent variables: $y^= b_{0} + b_{1} x_{1} + b_{2} x_{2}$ .

The response is total daily travel time; $x_{1}$ is the miles traveled and $x_{2}$ the number of deliveries. The coefficient of determination jumps: $R^{2} = 66.4%$ with the single variable $x_{1}$ , and about $90%$ with both $x_{1}$ and $x_{2}$ .

Worked example: R² with one versus two predictors (Butler tracking data, $n = 10$ ).

Model 1 — only miles traveled. The total sum of squares is $S S T = 23.900$ and the regression sum of squares is $S S R = 15.871$ , so:

$R^{2} = \frac{S S R}{S S T} = \frac{15.871}{23.900} = 0.6641 that is, 66.4%$

Model 2 — miles traveled plus number of deliveries. The same total $S S T = 23.900$ (it depends only on the data), but now $S S R = 21.6006$ :

$R^{2} = \frac{S S R}{S S T} = \frac{21.6006}{23.900} = 0.9038 that is, about 90%$

Step 3 — The adjusted R² for model 2, with $n = 10$ and $p = 2$ :

$R_{a}^{2} = 1 - (1 - 0.9038) \frac{10 - 1}{10 - 2 - 1} = 1 - (0.0962) \frac{9}{7} = 1 - 0.1237 = 0.8763$

Step 4 — Reading the outputs. The extra variable makes the explained variation larger — $S S E$ becomes smaller, so $R^{2} = S S R / S S T$ becomes larger — which indicates the suggested model is good. In comparison with simple regression, the multiple linear regression explains more (about 90% versus 66.4%), so better to go with the multiple linear regression for this problem.

Sense-check: adding deliveries closed most of the remaining error: $S S E$ dropped from $23.900 - 15.871 = 8.029$ to $23.900 - 21.6006 = 2.2994$ , a real gain. The adjusted $R^{2} = 0.8763$ is close to the raw 0.9038 because two predictors on ten observations is still a modest penalty — the two measures agree that the two-variable model is the better fit.

Visual intuition: picture the scatter of travel times against miles traveled: the one-variable line explains about two thirds of the vertical spread. Now imagine a second dimension — number of deliveries — lifting the points off the page; the two-variable model fits a tilted plane through the cloud, and the vertical gaps shrink dramatically. That tightening of the cloud onto the plane is the jump from 66.4% to about 90%.

Scope. Adjusted $R^{2}$ matters most when comparing models with different numbers of predictors — it is the honest comparator. Its formula assumes the same data set ( $n$ fixed) and the same dependent variable across the models being compared. Neither $R^{2}$ nor $R_{a}^{2}$ proves that the model is correct or that the variables are the right ones; they measure fit to the observed data. And when two predictors are strongly correlated (multicollinearity), even a high $R^{2}$ leaves the individual coefficients unstable — the model may be a good predictor while its coefficients cannot be trusted one at a time.

Pitfalls.

Celebrating every rise in $R^{2}$ . $R^{2}$ rises whenever a variable is added, useful or not. Judge additions with the adjusted $R^{2}$ — if $R_{a}^{2}$ does not improve, the new variable is not earning its place.
Keeping redundant predictors. If two independent variables are closely related to each other, there is no point keeping both — eliminate one. The second adds no fresh information and muddies the interpretation.
Judging from $R^{2}$ alone. The lecture's warning stands: you cannot conclude by looking at the R-square value alone; tests of significance follow and they carry the verdict.
Comparing $R^{2}$ across different data sets. The totals $S S T$ differ between data sets, so a raw comparison of $R^{2}$ values is not meaningful unless the data and the dependent variable are the same.

Exam note: know the three ideas of this section in one line each — adding independent variables raises $R^{2}$ ; adjusted $R^{2} = 1 - (1 - R^{2}) (n - 1) / (n - p - 1)$ accounts for the number of variables; multicollinearity (related independent variables) is the situation the adjusted $R^{2}$ is designed to address, and redundant variables should be dropped. In the Butler data: 66.4% with one predictor, about 90% with two.

Recap and bridge. Multiple linear regression $y^= b_{0} + b_{1} x_{1} + b_{2} x_{2} + \dots$ extends the single-line model to a plane; $R^{2}$ rises mechanically with every added variable, so the adjusted $R^{2}$ — which penalizes extra variables — and the multicollinearity check replace it as the honest fit measures. But even a great-looking $R^{2}$ is only a number: the next section asks whether the estimated model is statistically real, with the t-test for individual variables and the F-test for the overall model.

8.8 Tests of Significance: T-Test and F-Test

8.8.1 The Regression Model and the Beta-One Test

Hook: $R^{2} = 0.9027$ looks impressive — but how do you know the relationship is real rather than a coincidence of the sample? The estimated regression equation is only the first stage; tests of significance decide whether the model is actually appropriate.

The estimated regression equation is only the first stage. After estimating it, you must decide whether the assumed model is appropriate — whether the relationship is real. Write the simple linear regression model as

$y = β_{0} + β_{1} x + ε$

where $β_{0}$ is the intercept, $β_{1}$ is the slope (the change in $y$ per unit change in $x$ ), and $ε$ is the random error term.

The decisive question is about $β_{1}$ :

$H_{0} : β_{1} = 0 versus H_{1} : β_{1} \neq = 0$

If $β_{1} = 0$ , the model collapses to $y = β_{0}$ — the dependent variable does not depend on $x$ at all; $x$ has no influence. If $β_{1} \neq = 0$ , a relationship between $x$ and $y$ exists, and the regression has something to say. The hypothesis test that validates this is called the test for the significance of the regression.

What the tests need. Both the t-test and the F-test require an estimate of $σ^{2}$ , the variance of the error term $ε$ . The estimate comes from the residuals: $σ^{2}$ is estimated by the mean square error,

$s^{2} = M S E = \frac{S S E}{n - 2}$

where $n - 2$ is the degrees of freedom for simple linear regression (two parameters, $β_{0}$ and $β_{1}$ , were estimated to compute the residuals). Taking the square root gives $s = M S E$ , the standard error of the estimate. In the quarterly sales example, $s^{2} = M S E = 1530/8 = 191.25$ and $s \approx 13.83$ . The lecture takes the full discussion of this estimate up in the next session; for now it is enough to know that $M S E$ is the variance figure every test below divides by.

8.8.2 T-Test for Individual Variables and F-Test for the Model

Two tests do the validation work — not only the $R^{2}$ value:

The t-test checks the individual significance of each independent variable: whether a single coefficient like $β_{1}$ differs from zero, and more generally whether each variable in a multiple regression earns its place.
The F-test checks the overall significance of the model: whether the regression as a whole explains a significant amount of variation. The ANOVA table — the same style of table used for one-way and two-way ANOVA — is the vehicle for the F-test of the regression.

So the workflow is: estimate the regression equation, then run the tests of significance to decide the appropriateness of the assumed model, and only then recommend conclusions. The same t-and-F discussion applies inside the regression ANOVA table: both specifications are talked about there.

Formalize: the t-test for the slope. The least squares slope $b_{1}$ is a sample statistic; if you re-sampled the data you would get a somewhat different slope each time. Its sampling distribution is centered on the true $β_{1}$ , with a standard error estimated by

$s_{b_{1}} = \frac{s}{\sum _{i = 1}^{n} ( x _{i} - x ˉ ) ^{2}}$

The t-test statistic divides the estimated slope by its standard error:

$t = \frac{b _{1}}{s _{b_{1}}}$

which follows a t distribution with $n - 2$ degrees of freedom. Under the null $β_{1} = 0$ , a large $∣ t ∣$ (two-tailed test) means the estimated slope is too many standard errors from zero to be a coincidence, so $x$ really does influence $y$ . In a multiple regression, the same t-test runs once per coefficient $b_{i}$ , using $n - p - 1$ degrees of freedom, with the standard errors $s_{b_{i}}$ supplied by software.

Formalize: the F-test for the model. The F-test compares two independent estimates of $σ^{2}$ , exactly as in ANOVA. The regression sum of squares divided by its degrees of freedom gives the mean square due to regression:

$M S R = \frac{S S R}{p}$

with $p$ the number of independent variables (so $M S R = S S R$ in simple linear regression). The F statistic is

$F = \frac{M S R}{M S E}$

which follows an F distribution with $p$ numerator and $n - p - 1$ denominator degrees of freedom. A large F — right tail — rejects $H_{0} : β_{1} = 0$ (or in multiple regression, $H_{0} : β_{1} = β_{2} = \dots = β_{p} = 0$ ) and says the model as a whole explains a significant amount of variation.

Worked example: the significance tests on the quarterly sales regression.

Given: $y^= 60 + 5 x$ , $n = 10$ , $S S R = 14200$ , $S S E = 1530$ , $\sum (x_{i} - x ˉ)^{2} = 568$ , $s = 13.829$ .

T-test. The standard error of the slope is

$s_{b_{1}} = \frac{s}{\sum ( x _{i} - x ˉ ) ^{2}} = \frac{13.829}{568} = \frac{13.829}{23.83} = 0.5803$

and the test statistic is

$t = \frac{b _{1}}{s _{b_{1}}} = \frac{5}{0.5803} = 8.62$

With $n - 2 = 8$ degrees of freedom, $t = 8.62$ is far beyond the 0.005 mark (3.355), so the two-tailed P-value is below 0.01: reject $H_{0} : β_{1} = 0$ — population size is a significant predictor of sales.

F-test. The mean squares are

$M S R = \frac{S S R}{1} = 14200, M S E = \frac{S S E}{n - 2} = \frac{1530}{8} = 191.25$

$F = \frac{M S R}{M S E} = \frac{14200}{191.25} = 74.25$

With 1 and 8 degrees of freedom, F = 74.25 sits far in the right tail (the 0.01 mark is 11.26), so the P-value is below 0.01: reject the null — the regression as a whole is significant.

Sense-check: the two tests agree — $t = 8.62$ and $F = 74.25$ carry the same verdict, and in simple linear regression the F statistic is exactly the square of the t statistic: $8.6 2^{2} \approx 74.3$ . The high $R^{2} = 0.9027$ is now backed by formal evidence rather than a gut feeling.

Comparison: t-test versus F-test.

	t-test	F-test
Asks	Is one specific variable significant?	Is the model as a whole significant?
Null	$H_{0} : β_{i} = 0$	$H_{0} : β_{1} = \dots = β_{p} = 0$
Test statistic	$t = b_{i} / s_{b_{i}}$	$F = M S R / M S E$
Distribution	t with $n - p - 1$ df	F with $p, n - p - 1$ df
In simple regression	Verdict matches the F-test	Verdict matches the t-test
In multiple regression	One test per variable; unreliable under multicollinearity	The only test for overall significance

When to pick which: in simple linear regression either test suffices — they give identical verdicts; in multiple regression, use the F-test to establish that the model explains something, then the t-tests to see which variables earn their place.

The regression ANOVA table. The F-test lives in an ANOVA table exactly like the ones from Sections 8.1 and 8.4, with rows renamed for regression:

Source	Sum of squares	df	Mean square	F
Regression	SSR	p	MSR = SSR/p	F = MSR/MSE
Error	SSE	n - p - 1	MSE = SSE/(n - p - 1)
Total	SST	n - 1

The identity $S S T = S S R + S S E$ and the degrees-of-freedom split $n - 1 = p + (n - p - 1)$ are the same logic as ANOVA — the total variation of $y$ is carved up and each part compared against the noise floor.

Scope. The significance tests inherit the regression assumptions from Section 8.5: the error $ε$ has mean zero, constant variance, is independent, and is normally distributed. Those assumptions are what make $t = b_{1} / s_{b_{1}}$ and $F = M S R / M S E$ follow their claimed distributions. Rejecting $H_{0}$ proves only that $x$ and $y$ are related — it does not prove causation, and it does not prove the relationship is linear; a significant linear fit can still be a poor approximation of a curved truth. And under strong multicollinearity, individual t-tests can all say "not significant" while the F-test says the model is significant — the model works as a whole, but its coefficients cannot be separated into individual verdicts.

Pitfalls.

Concluding from $R^{2}$ alone. A high $R^{2}$ is suggestive but not proof; the tests of significance — not the fit measure — carry the verdict on whether the model is appropriate.
Mixing up the two tests. The t-test is for individual variables, the F-test for the overall model. Answering "is the model significant?" with a t-test (or "is $x_{1}$ significant?" with the F-test) is the classic swap.
Using the wrong degrees of freedom. t uses $n - 2$ (simple) or $n - p - 1$ (multiple); F uses $p$ and $n - p - 1$ . The 2 and the p are the estimated parameters.
Reading significance as causation. Rejecting $β_{1} = 0$ says the variables are associated; only domain judgment can turn that into a causal claim.

Exam note: for tests of significance, know the roles — the t-test validates the significance of individual independent variables, and the F-test validates the significance of the overall model. State the null $β_{1} = 0$ , compute or read the test statistic, compare the P-value with the level of significance, and comment on reject/accept in words.

Recap and bridge. After estimation, the model is validated by two tests: the t-test checks each coefficient against zero, and the F-test (through the regression ANOVA table) checks the model as a whole; both need the variance estimate $s^{2} = M S E = S S E / (n - 2)$ . In the sales example both tests decisively reject the null, confirming that population size really does drive quarterly sales. That closes the lecture's arc from ANOVA to regression — and the coming session wraps up the test for significance and revisits the whole syllabus for the mid-semester.

8.8.3 What Comes Next

The coming session wraps up the test for significance, then revisits the whole syllabus for the mid-semester, including the significance of the adjusted $R^{2}$ and multicollinearity, and how qualitative dependent variables are handled — in what way they differ from the existing regression models. A worked multiple linear regression using data analysis in Excel is planned too.

Exam Guidance Summary

ANOVA table interpretation in the mid-semester: you can expect the same style of work done in class — interpret the results of an ANOVA rather than build theory. Be ready to fill an incomplete ANOVA table when no specific data inputs are given; the simple layout (source, sum of squares, degrees of freedom, mean square, F, P-value) is the exact format to know. The arithmetic moves are always the same: $S S E = S S T - S S T R$ , $M S T R = S S T R / (K - 1)$ , $M S E = S S E / (N_{T} - K)$ , $F = M S T R / M S E$ .
Present classifications as ANOVA tables: both the one-way and two-way classifications are expected in the form of an ANOVA table — one-way with treatments and error; two-way with routes (or treatments), blocks (drivers), and error. Factorial designs are omitted from the syllabus — you can skip them.
P-values versus tables: the exam is open book, so tables are available. If you answer with P-values, you do not need the tables at all — compare the P-value with the level of significance and comment. Either path is accepted.
Software use: Excel or any other software may be used for problems; this was confirmed with the division team. Some problems still require manual work, and a mixture of both is fine.
Scope: the exam cannot go beyond what was delivered in class — roughly a portion, one question or something like that; because it is open book, a single question can differentiate the grades. The syllabus was notified and will be revisited for the mid-semester.
Multiple comparisons: after an ANOVA rejection, use the simple difference approach (compare the absolute difference of the two sample means with the LSD value) to locate which pair differs. No other multiple comparison procedure is required.
Coefficient of determination: expect a definition question — the percentage of variation in the dependent variable explained by the independent variable — and a numerical question computing $R^{2} = S S R / S S T$ with the percentage interpretation in words (e.g., "90.27% of the variability in sales is explained by the linear regression").
Tests of significance: the t-test validates individual independent variables; the F-test validates the overall model through the ANOVA table. Do not conclude from the $R^{2}$ value alone.
Regression relation: remember $S S T = S S R + S S E$ — the total variation, the explained variation, and the unexplained variation — and $R^{2} = S S R / S S T$ , which ranges between 0 and 1.
Quiz guidance: the solutions to quiz 1 are out, so there is no point re-taking it; attempt quiz 2 instead, available in the portal over a 15-day window from 16 to 30 September, with no time limit once you start. Practice problems and extra exercises will be shared, and a sample question session will be held to gauge readiness.

Key Industry Applications

Real-world: transit planning studies like the VARTA example — comparing mean travel time across bus routes — show how a two-way ANOVA with the driver as the second factor pulls variation out of the error term and sharpens the comparison.
Real-world: the unexplained-variation reading (routes explaining only about 31% of travel-time variation) is the standard diagnostic habit in industry: a huge error term signals that a factor is missing from the model.
Real-world: sales forecasting — quarterly sales predicted from population size with $y^= 60 + 5 x$ , where $R^{2} = 0.9027$ means about 90% of sales variability is explained by population size. Retail chains use exactly this kind of location-scoring regression to decide where to open outlets.
Real-world: the Butler tracking example — adding a second predictor lifts $R^{2}$ from 66.4% to about 90%, the classic demonstration of why analysts add explanatory variables (and why they then need adjusted $R^{2}$ to avoid being fooled by multicollinearity).
Real-world: Excel's data analysis tools for regression and ANOVA are accepted in place of hand computation — the same software that produces the F, t, and R² values in everyday statistical work.
Real-world: in practice, software prints P-values directly; the F and t distribution tables remain the fallback when only tables are available.

ASM Lecture 8 notes · Analysis of Variance and Regression

Advanced Statistical Methods· postgraduate· 2026-08-11

Sections Breakdown

18.1 One-Way Analysis of Variance and the ANOVA Table

Why ANOVA compares means through variance, the ANOVA table layout, filling an incomplete table, and equal versus unequal sample sizes.

28.2 Hypothesis Testing with the F-Distribution

The null and alternative hypotheses for ANOVA, the F test statistic, the Chemtech three-method worked example, and reading P-values from the F table.

38.3 Fisher's Least Significant Difference Test

When to use LSD after ANOVA rejects, the LSD formula and rejection rule, and completing pairwise comparisons.

48.4 Two-Way Analysis of Variance

The VARTA problem setup, the one-way treatment by routes, and the two-way decomposition into routes, drivers, and error.

58.5 Simple Linear Regression and the Decomposition of Variation

Best-fit lines and residuals, the quarterly sales example, and the SST = SSR + SSE decomposition of variation.

68.6 Coefficient of Determination

The definition of R-squared, the perfect-fit case, the worked example R-squared 0.9027, and interpreting R-squared in words.

78.7 Multiple Linear Regression, Adjusted R-squared, and Multicollinearity

Why adding variables raises R-squared, multicollinearity and the adjusted R-squared, and the Butler tracking worked example.

88.8 Tests of Significance: T-Test and F-Test

The regression model and the beta-one test, the t-test for individual variables, and the F-test for the model.

9Exam Guidance Summary

The professor's exam strategy: incomplete ANOVA tables, classifications as tables, open-book P-values, software use, and quiz guidance.

10Key Industry Applications

Real-world connections: VARTA transit planning, unexplained-variation diagnostics, sales forecasting, and the Butler tracking example.

Postgraduate students in statistics and data science

Exam Revision Notes

Below is the distilled, exam-ready core. Every entry comes from the full explanation above. Use this section for rapid review; return to the main notes when a point needs more context.

One-Way Analysis of Variance and the ANOVA Table

Must-know: SST = SSTR + SSE; df: K-1 treatments, N_T-K error, N_T-1 total; MSTR = SSTR/(K-1), MSE = SSE/(N_T-K), F = MSTR/MSE; be able to fill an incomplete ANOVA table.

$F = \frac{M S T R}{M S E} = \frac{S S T R / ( K - 1 )}{S S E / ( N _{T} - K )}$

⚠️ Top pitfall: Using the typo value 400 instead of the correct total sum of squares 460; miscomputing error degrees of freedom when sample sizes are unequal.

Self-check: For the five-method example (SSTR=300, SST=460, K=5, N_T=35), what are SSE, MSTR, MSE, and F?

Connects to: Hypothesis Testing with the F-Distribution; Fisher's Least Significant Difference Test; Two-Way Analysis of Variance.

Hypothesis Testing with the F-Distribution

Must-know: F is a right-tail test: reject H0 when the P-value (area right of computed F) is below alpha; for the Chemtech example F = 9.18 with (2,12) df, beyond the 0.01 mark 6.93, so P < 0.01 and the null is rejected.

$F = \frac{M S T R}{M S E} with (K - 1, N_{T} - K) degrees of freedom$

⚠️ Top pitfall: Claiming rejection means all means differ; the alternative says only that at least one mean differs.

Self-check: With (2,12) df the area right of 6.93 is 0.01. If F = 9.18, is the P-value below or above 0.01?

Connects to: One-Way Analysis of Variance and the ANOVA Table; Fisher's Least Significant Difference Test.

Fisher's Least Significant Difference Test

Must-know: LSD = t(alpha/2)*sqrt(MSE(1/n_i + 1/n_j)); reject H0 for pair (i,j) when |xbar_i - xbar_j| >= LSD; only run after ANOVA rejects. Chemtech: LSD = 2.179*sqrt(28.33*0.4) = 7.34; A-C difference 10 > 7.34 significant; A-B difference 4 < 7.34 not significant.

$L S D = t_{α /2} M S E (\frac{1}{n _{i}} + \frac{1}{n _{j}})$

⚠️ Top pitfall: Running LSD before ANOVA rejects the null; forgetting the absolute value in |xbar_i - xbar_j|.

Self-check: In the Chemtech example, why is there no significant difference between methods A and B?

Connects to: Hypothesis Testing with the F-Distribution; One-Way Analysis of Variance and the ANOVA Table.

Two-Way Analysis of Variance

Must-know: Two-way ANOVA: SST = SS_routes + SS_drivers + SSE with df k-1, b-1, (k-1)(b-1); error variation is less in the two-way analysis than in the one-way analysis. VARTA: routes explain 72.8 of 229.2, about 31%.

$S S T = S S_{routes} + S S_{drivers} + S S E$

⚠️ Top pitfall: Reading the 31% as 'routes are unimportant' instead of as a signal that a factor (drivers) is missing from the model.

Self-check: Why is SSE smaller in the two-way ANOVA than in the one-way ANOVA for the same data?

Connects to: One-Way Analysis of Variance and the ANOVA Table; Simple Linear Regression and the Decomposition of Variation.

Simple Linear Regression and the Decomposition of Variation

Must-know: Estimated regression y-hat = 60 + 5x from ten observations with xbar = 14, ybar = 130, b1 = 2840/568 = 5, b0 = 130 - 5(14) = 60; SST = SSR + SSE with SST = 15730, SSR = 14200, SSE = 1530.

$S S T = S S R + S S E, S S T = \sum (y_{i} - y ˉ)^{2}, S S E = \sum (y_{i} - y^_{i})^{2}, S S R = \sum (y^_{i} - y ˉ)^{2}$

⚠️ Top pitfall: Confusing the total deviation y_i - ybar with the residual y_i - yhat_i; forgetting that without x the best prediction is ybar.

Self-check: Why does the regression line always pass through the point (xbar, ybar)?

Connects to: Coefficient of Determination; Tests of Significance: T-Test and F-Test; One-Way Analysis of Variance and the ANOVA Table.

Coefficient of Determination

Must-know: R^2 = SSR/SST = 1 - SSE/SST lies in [0,1]; perfect fit (yi = yhat_i) gives SSE = 0, SST = SSR, R^2 = 1; quarterly sales: R^2 = 14200/15730 = 0.9027, about 90.27%.

$R^{2} = \frac{S S R}{S S T} = \frac{14200}{15730} = 0.9027$

⚠️ Top pitfall: Concluding from the R^2 value alone; R^2 is SSR/SST, never SSE/SST.

Self-check: If every observed value equals its predicted value, what are SSE and R^2?

Connects to: Simple Linear Regression and the Decomposition of Variation; Multiple Linear Regression, Adjusted R-squared, and Multicollinearity; Tests of Significance: T-Test and F-Test.

Multiple Linear Regression, Adjusted R-squared, and Multicollinearity

Must-know: R^2 rises as variables are added; adjusted R^2 = 1 - (1-R^2)(n-1)/(n-p-1) compensates; multicollinearity (|correlation| > 0.7 between predictors) means drop one of the related pair; Butler data: 66.4% with one predictor, about 90% with two.

$R_{a}^{2} = 1 - (1 - R^{2}) \frac{n - 1}{n - p - 1}$

⚠️ Top pitfall: Concluding a model is better purely from a higher R^2; keeping two variables that measure essentially the same thing.

Self-check: Why can R^2 never fall when an independent variable is added?

Connects to: Coefficient of Determination; Tests of Significance: T-Test and F-Test; Simple Linear Regression and the Decomposition of Variation.

Tests of Significance: T-Test and F-Test

Must-know: t-test validates individual variables (t = b1/s_b1, n-2 df); F-test validates the overall model (F = MSR/MSE, p and n-p-1 df) via the ANOVA table; beta1 = 0 collapses the model to y = beta0 so x has no influence. Sales example: t = 8.62, F = 74.25, both reject H0.

$t = \frac{b _{1}}{s _{b_{1}}}, F = \frac{M S R}{M S E}$

⚠️ Top pitfall: Concluding from R^2 alone instead of running the significance tests; using the t-test for the model and the F-test for a single variable.

Self-check: What does beta1 = 0 imply about the regression model y = beta0 + beta1 x + epsilon?

Connects to: Simple Linear Regression and the Decomposition of Variation; Coefficient of Determination; Multiple Linear Regression, Adjusted R-squared, and Multicollinearity; One-Way Analysis of Variance and the ANOVA Table.

Previous LectureLecture 07 · Design of Experiments, Analysis of Variance, and Regression Analysis

Next LectureLecture 09 · Regression Analysis: Significance Testing, Multiple Regression, and Categorical Predictors

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Prerequisite Knowledge

Previously Covered in This Subject

8.1 One-Way Analysis of Variance and the ANOVA Table

8.1.1 Why Analysis of Variance Exists

8.1.2 The ANOVA Table Layout

8.1.3 Worked Example: Filling an Incomplete ANOVA Table

8.1.4 Equal and Unequal Sample Sizes

8.2 Hypothesis Testing with the F-Distribution

8.2.1 The Hypotheses

8.2.2 Worked Example: The Chemtech Three-Method Problem

8.2.3 Reading P-Values from the F Table Without Computing Them

8.3 Fisher's Least Significant Difference Test

8.3.1 When to Use LSD (and When Not To)

8.3.2 The LSD Formula and Rejection Rule

8.3.3 Worked Example: Method A versus Method C

8.3.4 Completing the Pairwise Comparisons

8.4 Two-Way Analysis of Variance

8.4.1 The VARTA Problem Setup

8.4.2 One-Way Treatment: Routes Only

8.4.3 Two-Way Decomposition: Routes, Drivers, and Error

8.5 Simple Linear Regression and the Decomposition of Variation

8.5.1 Recap: Best-Fit Lines and Residuals

8.5.2 The Quarterly Sales Example

8.5.3 SST, SSR, and SSE: Three Sources of Variation

8.6 Coefficient of Determination

8.6.1 Definition

8.6.2 What Happens When There Is No Error

8.6.3 Worked Example: R² = 0.9027 for Quarterly Sales

8.6.4 Interpreting R² in Words

8.7 Multiple Linear Regression, Adjusted R², and Multicollinearity

8.7.1 Adding Variables Raises R²

8.7.2 Multicollinearity and the Adjusted R²

8.7.3 Worked Example: Butler Tracking Data

8.8 Tests of Significance: T-Test and F-Test

8.8.1 The Regression Model and the Beta-One Test

8.8.2 T-Test for Individual Variables and F-Test for the Model

8.8.3 What Comes Next

Exam Guidance Summary

Key Industry Applications

ASM Lecture 8 notes · Analysis of Variance and Regression

Sections Breakdown

Exam Revision Notes

One-Way Analysis of Variance and the ANOVA Table

Hypothesis Testing with the F-Distribution

Fisher's Least Significant Difference Test

Two-Way Analysis of Variance

Simple Linear Regression and the Decomposition of Variation

Coefficient of Determination

Multiple Linear Regression, Adjusted R-squared, and Multicollinearity

Tests of Significance: T-Test and F-Test

Was this lecture useful?

BitsNotes AI Assistant

Configure AI Chat

Sign in to use AI Chat