Design of Experiments, Analysis of Variance, and Regression Analysis

This session has two parts. The first part introduces the design of experiments — how we plan an experiment so we can see what actually affects the final outcome — and then the analysis of variance (ANOVA), the statistical procedure that analyzes the specific design. The second part introduces regression analysis, in both its simple linear and multiple linear forms. The reference chapters are chapter 13 (design of experiments and analysis of variance) and chapter 14 (regression). The plan is to spend the first part of the time on the fundamental, basic design, and to take a few examples to address both topics broadly.

7.1 Design of Experiments

7.1.1 What the Design of Experiments Is About

What if you could know, before running a single trial, which ingredients of an experiment will actually move the final result? That is the question the design of experiments (DOE) answers. It is the deliberate planning of an experiment so that the effects we care about are visible and the effects we do not care about are kept quiet — a skill that decides whether an experiment teaches you something or merely gives you a pile of numbers.

The effect itself is what we call the amount of variation. As far as the interior design is concerned, that variation will be affected by the individual factors, or by multiple factors, or by the combinations of the factors considered together — and that last possibility is a very important point. So what we are trying to do throughout is figure out how we are going to capture the variation, the total variation. If we want to address the total variation in a particular problem, we need to analyze it, and that is exactly what the design of experiments is for: once the design is identified, we can manipulate multiple input factors and determine their effects on the desired output. If a design does not give you a proper result in terms of your output, you think about another possible combination — whether you need to include another factor or not — and all of that comes into the picture.

Think of total variation as a fixed budget of "noise" that every experiment has. Good design spends as much of that budget as possible on factors you can explain, and leaves as little as possible to pure chance. The design of experiments is the planning that decides, before data collection, how the budget is split.

7.1.2 Types of Designs and the Basic Principles

As far as the types of designs are concerned, there are single-factor designs, where you consider one factor; multiple-factor designs, where you consider several factors; and interaction-factor designs, where the interaction between factors matters. These are the types of designs available to us, and it is up to us which design we recommend for a given problem.

For the sake of the discussion, the basic recommended designs are:

• Completely randomized design — the simplest fundamental design. Every treatment is assigned to the experimental units entirely at random; in the Chemtech example coming up, the three assembly methods are assigned at random to 15 workers, five per method.
• Randomized block design — you divide the experimental units into blocks (groups of "similar" units) and then randomize the treatments within each block. Blocking removes the variation that comes from the differences between blocks — for example, testing three fertilizers on soil of different quality by putting all three on each soil type.
• Factorial design — you study two or more factors together, including their combinations. Here the interaction between factors can be detected — something a one-factor-at-a-time approach can never reveal.

There are many more advanced designs, but the discussion here is confined to these fundamental ones; the subsequent things follow in a routine fashion once the basic design is understood.

The three designs are not competing recipes; they are a ladder of control. The completely randomized design is the baseline, the randomized block design adds control over a known source of variation, and the factorial design adds the ability to see combinations working together.

Real-world: the completely randomized design, the randomized block design, and the factorial design are the workhorses of industrial experiments — agriculture, manufacturing, pharmaceutical trials and product testing all run on these three basic structures before any of the advanced designs are invoked.

7.1.3 ANOVA as the Tool for the Design

At a low level, the relationship between the two ideas can be put like this: the analysis of variance is the tool — like the scissors a tailor uses — and the design of experiments is what we want to build with that tool. For all the things a proper design needs, you need some specific tools, and that tool is the analysis of variance, ANOVA for short.

The need for such a tool is real. We know that the significance of variance is everywhere, but at this point the question is how to analyze it: at times it is very difficult to analyze the total variation. You can come up with the total variation, but you do not know which factors are influencing it more and which factors are contributing less. Analysis of variance is one such statistical tool that helps us address exactly that.

The same basic intention appears in regression, which is discussed later in the session: your main intention is to understand the effect of the output. In our language, we call them dependent variables and independent variables — dependent variables are the response variables, and independent variables are the input factors. With the help of those independent variables, we study how the output is affected. Even more precisely, in regression we will get to know which factors are significantly contributing and which are not; if they are not significantly contributing, we try to eliminate them and remodel the problem.

To do that, one way alone may not make sense — you ask whether you can include any other possible factors, you include them, and you observe. If you go on refining your designs, your chance causes will diminish. Each added factor or interaction is a transfer of variation out of the "chance" column and into the "explained" column.

7.1.4 Exam Notes

Exam note: chapter 13 is the reference for this design of experiments and analysis of variance discussion. The chapter carries a full set of worked example studies — the Chemtech filtration-system assembly experiment, which we work through in the next section, and an observational study at National Computer Products comparing quality-awareness exam scores across three plants, among others. The further reading for anyone interested is the book Design of Experiments by Montgomery, but the syllabus and the textbook portion cover only the fundamental designs, and only what is relevant from the exam point of view is discussed.

7.2 From Two Means to Many: Why ANOVA Exists

7.2.1 The Two-Sample Comparison and the t-Test

Suppose you want to know whether two types of music change the mean performance of a mental task. You have two groups of people, two sample means, and one question: is the difference real, or just luck? This is the familiar territory of the Z test and the t test — the tools you already know — extended from a single population to a comparison.

Most of you are already familiar with the Z test and the t test. In that discussion, we worked with a single population and tried to validate logical statements about it. The extension here comes when a comparison is involved. Suppose an experiment aims to determine whether two types of music have different effects on the mean performance of a mental task. Now we are talking about population one and population two, and we want to make a judgment on which one is better — whether population one's performance is better than population two's and so on. We are no longer testing a single mean; we are testing whether the two means are equal:

$$H_0:{\mu }_1={\mu }_2\text{against}{H}_A:{\mu }_1\ne {\mu }_2$$

The two-sided alternative is the case being considered, but depending on the context it may be greater than $({\mu }_1>{\mu }_2)$ or less than $({\mu }_1<{\mu }_2)$. To validate this, you again use the t test or the Z test — for the sake of discussion here we go with the t test. You calculate the test statistic and the p-value; if the p-value is less than alpha, you reject the null, and so on. This is all routine business; these two-mean cases are not going to be asked in the exam by themselves — they are introduced only to motivate what comes next.

7.2.2 Three or More Means: The Pairwise t-Test Problem

Now suppose we are interested in assessing the relative effects of three types of music, or four types of music. It is no longer a comparison of two; if it is more than two, we write

$$H_0:{\mu }_1={\mu }_2={\mu }_3\text{against}{H}_A:\text{not all population means are equal}$$

At first glance this looks like a simple extension. But when you want to validate it, if you stick with the t test you need to consider two populations at a time. You ask whether there is a significant difference between one and two, between two and three, between one and three — and so on. With two population means you need one simple t test; when it extends from two to three, there is a set of combinations of t tests to perform. If you extend the idea to five populations at a time — ${\mu }_1={\mu }_2={\mu }_3={\mu }_4={\mu }_5$ — the number of t tests increases accordingly: you take all combinations of the populations two at a time and validate each one. This is a tedious, complicated thing to expect: the number of pairings grows quickly, and when the number of populations increases it becomes very difficult to list out all the possible t tests and comment on the situation.

How quickly does the number of tests grow? The count of pairwise comparisons among $k$ populations is the number of ways to choose 2 populations out of $k$, written $\left(\genfrac{}{}{0ex}{}{k}{2}\right)$:

$$\left(\genfrac{}{}{0ex}{}{k}{2}\right)=\frac{k(k-1)}{2}$$

So three populations already demand three separate t tests, and five populations demand ten. And each individual test has its own chance of error — every "reject" you check inflates the overall chance of a false conclusion. This is why a different idea is needed.

To avoid these difficulties, instead of doing the pairwise t tests two at a time, the idea is: what if we consider the variation among these situations instead of talking in terms of means? If we take the variation, we try to find out how best it influences the problem by capturing the variation — and with a single captured quantity we can address five, six, seven populations at once. That is the concept of the analysis of variance. At its lowest level, ANOVA is an essential extension of the logic of the t test to those situations where we wish to compare the means of three or more populations: whenever more than two population means are to be considered concurrently, the analysis of variance is the better, recommended approach.

7.2.3 The Variation Viewpoint: What Small and Large Variability Mean

Consider the hypothesis $H_0:{\mu }_1={\mu }_2={\mu }_3$ and $H_A:$ not all population means are equal. Think about what the world looks like if the null hypothesis is true. The picture used here is the sampling distribution of the sample means: you know that $\bar{x}$ follows a normal distribution (assuming, as we do, that the population comes from a normal distribution). If the null is true — the three population means are equal — then the sample means ${\bar{x}}_1,{\bar{x}}_2,{\bar{x}}_3$ will also be close together. The closer the three sample means are, the weaker is the evidence we have to reject the null hypothesis; the more the sample means spread apart, the stronger is the evidence to reject. That is the conclusion in terms of averages.

But the approach here is not to inspect the sample means directly — it is to look at the variability. There are two possibilities: the variability among the sample means may be small, or the variability among the sample means may be large. If the variability among the sample means is small, which hypothesis does common sense tell us to support? The null hypothesis — because when the null is true the means are equal, so their variation is small. If the variability is large, it supports the alternative. That is the whole reasoning being established here.

To make this quantitative we need two things: the population mean $\mu$ and the population variance ${\sigma }^2$. The population mean is unknown, so we replace it with the grand mean, written as $\overline{\stackrel{ˉ}{x}}$ ("x double bar"), because we have three sample means ${\bar{x}}_1,{\bar{x}}_2,{\bar{x}}_3$ and we need one reference point. The population variance ${\sigma }^2$ is also unknown, so we introduce the variability through two best estimates. When the null hypothesis is true, we anticipate one particular variability: the variation between the sample means, which we call the variability between the treatments (between the methods). The between-treatments variation will give a reliable estimate of the population variance when the null hypothesis is true. When the alternative hypothesis is true, the situation is different: if you go with the variation between the treatments under the alternative, your estimate of ${\sigma }^2$ will overestimate. So instead of relying on the between variation, we turn to the within variation: we find the variation within the treatments and use that. The rule of thumb as stated here: under the null hypothesis you calculate the variation between the treatments; under the alternative hypothesis you calculate the variation within the treatments.

The whole thing rests on one decomposition: the total variation can happen between the treatments or within the treatments. You are bifurcating the total variation, decomposing it into the variation between the methods and the variation within the methods. The between-treatments variation corresponds to the assignable causes we control, and the within-treatments variation is the chance or error component.

Visualize it as two histograms side by side. On the left, the sampling distribution of the sample means when the null is true: one bell curve centered at $\mu$, and the three sample means ${\bar{x}}_1,{\bar{x}}_2,{\bar{x}}_3$ sitting close together near the middle — close, so small between-variation, weak evidence against $H_0$. On the right, the null false: three separate bells centered at different ${\mu }_1,{\mu }_2,{\mu }_3$, and the three sample means scattered far apart — spread out, so large between-variation, strong evidence against $H_0$. The one-sentence takeaway: the spread of the sample means is the evidence; the width of each individual bell is the noise we must compare it against.

7.2.4 Recap: From Two Means to Many

The chain of reasoning for this section: the t test handles two means; more means means many pairwise tests; many tests is tedious and error-prone; so we compare one number (between-treatment spread) with another number (within-treatment spread). The section that follows shows the exact formulas for those two numbers and the F test that compares them, on the Chemtech filtration system example.

7.3 One-Way ANOVA: The Chemtech Filtration System Example

7.3.1 The Problem Setup

The example is the fundamental one from the textbook. A company (Chemtech) wants to develop a new filtration system, and the interest is in observing the response of this new filtration system — which is the dependent variable in our language. The components for the new filtration system will be purchased from several suppliers, and Chemtech will assemble the components at its plants in Columbia and South Carolina. The industrial engineering group is responsible for determining the best assembly method for the new filtration system. Here the assembly method is the independent variable, or the factor that influences the filtration system: different assembly methods will affect the new filtration system differently. The company is interested in validating whether there is any significant difference among the three methods they are suggesting — method A, method B and method C — and, in the end, in determining which assembly method can produce the greatest number of filtration systems.

This is a textbook case of the completely randomized design from the previous section: one factor (assembly method), three treatments (the methods), and a quantitative response (units produced per week). The three methods define three populations — all employees who would use method A, all who would use method B, all who would use method C — and the statistical question is whether the mean production of those three populations is the same.

7.3.2 Randomization and the Data

The schematic setup is simple. The population of interest is the employees at the plants in the different places mentioned. From that population, 15 employees were randomly selected. The purpose of using randomness is that no personal bias should come into addressing the problem — randomization is one of the principles of any specific design, along with replication, which reduces the amount of bias within our control. Each of the three assembly methods is then randomly assigned to five employees: method A to five employees, method B to five, method C to five, and they work it out to see which one is better and whether the difference is statistically significant. The data considered are the number of units assembled by each employee during one week, shown in the table below. Because each method is used by five workers, we say the experiment has five replicates per treatment.

The notation, with the definitions the example will use:

• ${\mu }_j$ — the mean of the $j$-th population (here, the mean number of units produced per week by method $j$).
• $x_{ij}$ — the observation of the $i$-th observation for treatment $j$; so we have $x_{11},x_{12},x_{13},x_{21}$, and so on.
• $n_j$ — the number of observations in treatment $j$: for method A there were five observations, similarly for B and C.
• ${\bar{x}}_j$ — the sample mean of treatment $j$.
• $s_j^2$ — the sample variance of treatment $j$.
• $n_T$ — the total sample size: $5+5+5=15$.

There are different cases to keep in mind: one with equal sample sizes, as here, and the other with unequal sample sizes. All the formulas in this section handle both, but the arithmetic in the worked example is simplest in the equal-size case.

7.3.3 The Hypotheses

Let ${\mu }_1$ be the mean number of units produced per week by method A, ${\mu }_2$ by method B, and ${\mu }_3$ by method C. At the population level we want to verify

$$H_0:{\mu }_1={\mu }_2={\mu }_3\text{against}{H}_A:\text{not all population means are equal}$$

That is the real objective: whether the three sample means observed are different enough for us to conclude that the population means corresponding to the three assembly methods are different. One way to address the problem is to perform the three pairwise combinations of t tests and come to a conclusion; the other way, the one we take, is to capture the amount of variation and see whether it makes sense.

Notice what the alternative does not say: it does not claim all three means differ — only that at least two of them do. If the null is rejected, the ANOVA itself does not tell us which pair differs; pinning that down is the job of follow-up procedures such as Fisher's LSD, covered in later sessions.

7.3.4 The Sample Mean and Variance Formulas

The formulas are the standard ones you already know: "$\bar{x}$ is summation xi by n" and "$s^2$ is summation xi minus x bar whole square by n minus 1." For each treatment $j$:

$${\bar{x}}_j=\frac{{\sum }_{i=1}^{n_j}{x}_{ij}}{n_j},s_j^2=\frac{{\sum }_{i=1}^{n_j}(x_{ij}-{\bar{x}}_j{)}^2}{n_j-1}$$

To get ${\bar{x}}_j$ we add all five observations of that method and divide by 5. The sample means for the three methods come out as ${\bar{x}}_1=62$ (method A), ${\bar{x}}_2=66$ (method B), and ${\bar{x}}_3=52$ (method C). Quick check on method A: $(58+64+55+66+67)/5=310/5=62$. For method C: $(48+57+59+47+49)/5=260/5=52$. The corresponding sample variances $s_1^2=27.5$, $s_2^2=26.5$, $s_3^2=31.0$ are computed from the squared deviations around each sample mean. For method A, the five deviations from 62 are $-4,+2,-7,+4,+5$; their squares are $16,4,49,16,25$, summing to 110, and

$$s_1^2=\frac{(58-62{)}^2+(64-62{)}^2+(55-62{)}^2+(66-62{)}^2+(67-62{)}^2}{5-1}=\frac{110}{4}=27.5$$

The same calculation for method B gives $s_2^2=26.5$ and for method C gives $s_3^2=31.0$. The pooled quantity SSE below uses exactly these variances and lands on the published value 340, which is the check that matters.

From the three sample means we get the grand mean, the overall mean, the reference point that replaces the unknown population mean $\mu$:

$$\overline{\stackrel{ˉ}{x}}=\frac{{\bar{x}}_1+{\bar{x}}_2+{\bar{x}}_3}{3}=\frac{62+66+52}{3}=60$$

The verbal description: "X double bar here is a grand mean... ${\bar{x}}_1+{\bar{x}}_2+{\bar{x}}_3$ divided by 3, which is turned out to be 60." For equal sample sizes this simple average of the three sample means equals the overall average of all observations — if the sizes were unequal, the grand mean would instead be the weighted average $\overline{\stackrel{ˉ}{x}}=\sum n_j{\bar{x}}_j/n_T$. Under the null hypothesis, this grand mean of 60 is the best single estimate of the common population mean $\mu$.

7.3.5 Between-Treatments Variation: SSTR and MSTR

The between-treatments estimate of the population variance is the mean square due to treatments, abbreviated MSTR, built from the sum of squares due to treatments, SSTR. Here the treatments are the three methods: method A, method B and method C, so $K=3$, and the sum runs over $j=1$ to $K$:

$$SSTR=\sum _{j=1}^K{n}_j({\bar{x}}_j-\overline{\stackrel{ˉ}{x}}{)}^2$$

The verbal description: "the formula... $n_j$ times ${\bar{x}}_j$ minus $\overline{\stackrel{ˉ}{x}}$ whole square." This is the same structure as the simple variance formula $\sum (x-\mu {)}^2$, with each sample mean ${\bar{x}}_j$ playing the role of $x$ and the grand mean playing the role of $\mu$, weighted by the number of observations in each treatment. Substituting:

$$SSTR=n_1({\bar{x}}_1-\overline{\stackrel{ˉ}{x}}{)}^2+n_2({\bar{x}}_2-\overline{\stackrel{ˉ}{x}}{)}^2+n_3({\bar{x}}_3-\overline{\stackrel{ˉ}{x}}{)}^2$$

$$SSTR=5(62-60{)}^2+5(66-60{)}^2+5(52-60{)}^2=5(4)+5(36)+5(64)=20+180+320=520$$

With $K=3$ treatments, the degrees of freedom are $K-1=2$, and the mean square due to treatments is

$$MSTR=\frac{SSTR}{K-1}=\frac{520}{2}=260$$

So the between-treatments variation captured by this formula is MSTR = 260, the estimate we rely on when the null hypothesis is true. Why divide by $K-1$? The three sample means ${\bar{x}}_1,{\bar{x}}_2,{\bar{x}}_3$ are free to vary only in $K-1=2$ independent directions once the grand mean 60 is fixed — knowing any two of them forces the third, since their average must stay 60.

7.3.6 Within-Treatments Variation: SSE and MSE

The within-treatments estimate of the population variance is the mean square due to error, MSE, built from the sum of squares due to error, SSE — also called the sum of squares due to chance, in line with the chance-causes view of the within variation. The formula pools the sample variances of the treatments:

$$SSE=\sum _{j=1}^K(n_j-1)s_j^2$$

With $n_1=n_2=n_3=5$, each term is $4s_j^2$, and the computed value is

$$SSE=(5-1)(27.5)+(5-1)(26.5)+(5-1)(31.0)=4(27.5)+4(26.5)+4(31.0)=110+106+124=340$$

So SSE = 340. The degrees of freedom within treatments are $n_T-K=15-3=12$, so

$$MSE=\frac{SSE}{n_T-K}=\frac{340}{12}=28.33$$

When the sample sizes are equal, MSE is simply the average of the three sample variances: $(27.5+26.5+31.0)/3=85/3=28.33$. So the complete variation is now decomposed into two pieces: the variation due to the methods, numerically captured as 260 (between the methods), and the variation within the methods, captured as 28.33.

Note the contrast between the two estimates: 260 is over 9 times 28.33. If the null hypothesis were true, both numbers would be estimates of the same ${\sigma }^2$ and would sit close together, near a ratio of 1. A ratio this large is the first signal that the means are not equal — the formal test is next.

7.3.7 The F Statistic and the Decision

Whenever we want to compare two variations — here the between-treatments variation against the within-treatments variation — the comparison is of the form ${\sigma }_1^2/{\sigma }_2^2$, and the distribution that governs the comparison is the F distribution (the discussion does not go into the F distribution tables in detail; the p-value route is used). The F statistic is

$$F=\frac{MSTR}{MSE}=\frac{260}{28.33}=9.18$$

Always remember the structure: "MSTR by MSE." The MSE always sits in the denominator; the reason for that placement is a concept that will be explained later, not now. (Preview of the reasoning: MSE is unbiased whether the null is true or not, so it is the natural yardstick; MSTR is the number that inflates when the means differ, so the ratio grows exactly when there is something to detect.) The degrees of freedom are 2 in the numerator and 12 in the denominator.

The decision logic is the usual p-value logic. If $p\le \alpha$, reject the null hypothesis; if $p>\alpha$, accept the null hypothesis. For the portion of the F table with 2 and 12 degrees of freedom, the right-tail area is examined: for an F value of about 6.93, the area to the right is 0.01. Our F value of 9.18 lies beyond 6.93, so the area to its right must be less than 0.01 — the p-value is below 0.01, well below $\alpha =0.05$. Without any doubt the p-value is less than alpha, so we reject the null hypothesis and conclude that not all population means are equal. When you run this in Excel or Minitab, you can witness the exact p-value directly — the textbook reports the exact value as 0.004 — and verify leisurely what the value turns out to be.

Visual intuition: draw the F distribution for $d{f}_1=2,d{f}_2=12$ — a right-skewed curve starting at 0, rising to a low peak near 1, and trailing off to the right. The p-value is the area under the curve to the right of the observed $F=9.18$, which sits far out in the tail. The one-sentence takeaway: the farther the ratio sits in the right tail, the less plausible it is that both estimates came from the same variance — and the stronger the evidence against equal means.

7.3.8 The ANOVA Table

Always expect the analysis of variance to be presented as a table — this is how Excel and most software report ANOVA output. The first column stands for the source of variation, the second column stands for the sum of squares, the third column stands for the degrees of freedom, the fourth column stands for the mean square, and then come the F statistic and the p-value. The source of variation for this single-factor problem has three rows: methods (the one factor being considered), error, and total.

The total variation is the sum of squares due to the total, SST, and the identity always holds:

$$SST=SSTR+SSE$$

The verbal statement: "It is always true... the total variation is now decomposed into the variation between the methods and the variation within the methods." So $SST=520+340=860$, with no doubt. The same additivity holds for the degrees of freedom: $2+12=14$, and equivalently $15-1=14$, since the total degrees of freedom are $n_T-1$. The computation in full: SSTR = 520 with $K-1=2$ degrees of freedom, MSTR = 260; SSE = 340 with $n_T-K=12$ degrees of freedom, MSE = 28.33; and the F statistic $9.18$ with numerator 2 and denominator 12.

7.3.9 Student Questions and Answers

7.3.10 Exam Notes

Exam note: if an analysis of variance question is asked, instead of reporting the information here and there, the ANOVA is always tabulated — source of variation, sum of squares, degrees of freedom, mean square, F statistic and p-value — exactly the picture that Excel and software give you. Remember the identity $SST=SSTR+SSE$, the degrees-of-freedom arithmetic ($K-1$, $n_T-K$, $n_T-1$), the F statistic $MSTR/MSE$ with MSE in the denominator, and the p-value decision rule $p\le \alpha$ reject, $p>\alpha$ accept. The F p-value is read from the right tail, like the Z, t and chi-square p-values. All the detail is in the book — chapter 13 covers this discussion; every value shown here matches what you can look up there.

7.4 Simple Linear Regression

7.4.1 From ANOVA to Regression

The regression discussion shares the same basic intention as ANOVA: understanding the effect on the output. In our language, dependent variables are the response variables and independent variables are the input factors. With regression we can go one step further and pinpoint which factors are significantly contributing and which are not; if some are not significantly contributing, we eliminate them and remodel the problem. The first building block is simple linear regression — one independent variable and one dependent variable, both continuous.

So the thread across the session is: design of experiments plans the experiment, ANOVA analyzes whether a categorical factor's levels differ, and regression analyzes the effect of continuous input variables on a continuous output. The same vocabulary — dependent variable, independent variable, significant contribution — runs through all three.

7.4.2 The Model and What the Slope Means

Given a set of observed points $(x_i,y_i)$, we want to find one particular line that comes close to the points. If the observed value and the expected value were the same, the error would be minimum — but a zero error is not possible practically; what we try to achieve is that the expected value of the error is 0. The line is written

$$\hat{y}=b_0+b_{1}x$$

where ${\hat{y}}_i$ is the estimated value of the dependent variable for the $i$-th observation, $y_i$ is the observed value, $b_0$ (B0) and $b_1$ (B1) are the best replacements for the population parameters ${\beta }_0$ and ${\beta }_1$ of the relation ${\beta }_0+{\beta }_{1}x$, and $x$ is the independent variable. The meaning of the slope: if ${\beta }_1$ is positive, there is a positive linear relationship; if ${\beta }_1$ is negative, there is a negative linear relationship; if ${\beta }_1$ is 0, there is no relation existing between $x$ and $y$. These are the three possible combinations.

The y-intercept $b_0$ is the point where the line crosses the y-axis (the value of $\hat{y}$ when $x=0$), and the slope $b_1$ is the expected change in $y$ for a one-unit increase in $x$. If $b_1=5$, then for every extra unit of $x$, $y$ is expected to rise by 5 units — this number is the practical answer regression gives.

7.4.3 The Principle of Least Squares

How do we pick the one best line out of infinitely many? Plot the observed values $y_i$ against the fitted line: there is a deviation between each observed point and the line — some points sit on the line, others deviate from it. The moment we find values of $b_0$ and $b_1$ such that the error is minimum, we have the best fitted line. That is the principle of least squares: the sum of squares of the residuals is always minimum. The residual is $y_i-{\hat{y}}_i$, so we minimize

$$\underset{b_0,b_1}{\min }\sum _{i=1}^n(y_i-b_0-b_1{x}_i{)}^2$$

The verbal description: "summation i equals 1 to n of yi minus B0 minus B1x whole square — this is the absorbed value and this is the expected value; this total is my residual." To find the minimizing values we use calculus: take the partial derivative with respect to $b_0$ and equate it to 0, which gives one formula for $b_0$; take the partial derivative with respect to $b_1$ and equate it to 0, which gives one formula for $b_1$. Setting the derivatives equal to zero picks out one best possible combination from infinitely many numbers — the line closest to the observed values, the one with least error.

Why squares and not plain errors? If we minimized $\sum (y_i-{\hat{y}}_i)$, positive and negative deviations would cancel each other — a line that misses every point by a lot could still score zero. Squaring makes every deviation count, punishes large misses extra hard, and makes the calculus clean. That is the whole reason the criterion is the sum of squared residuals.

7.4.4 Worked Example: Computing the Best-Fit Line

The example is the trucking problem (the name is garbled in the recording as "auto periodic"; the point is the computation itself). The given (or computed) values are $\bar{x}=2$ and $\bar{y}=20$, where $\bar{x}=\sum x_i/n$ and $\bar{y}=\sum y_i/n$. If a calculation is asked in the exam, $x_i$ and $y_i$ will be given, and you compute the four deviation columns: $x_i-\bar{x}$, $(x_i-\bar{x}{)}^2$, $y_i-\bar{y}$, and $(x_i-\bar{x})(y_i-\bar{y})$.

Among infinitely many lines there is no other combination like $11+3x$ or $14+7x$: $10+5x$ is the best one because it takes care of the minimum error, the smallest possible error. And for any value of $x$ you can predict $y$: substitute the value of $x$ into $\hat{y}=10+5x$ and compute. For example, at $x=2.5$ the prediction is $\hat{y}=10+5(2.5)=22.5$.

Visual intuition: draw the scatter plot with $x$ on the horizontal axis and $y$ on the vertical axis; the five points cluster around an upward diagonal. The fitted line $y=10+5x$ cuts through the cloud, with some points above it and some below. The vertical distance from each point to the line is its residual; the best line is the one that makes the sum of the squared vertical distances as small as possible.

7.4.5 Student Q&A: How Is the Best Line Chosen?

7.4.6 Exam Notes

Exam note: in calculation-oriented questions, $x_i$ and $y_i$ will be given; compute $\bar{x}$ and $\bar{y}$, then the four deviation columns $x_i-\bar{x}$, $(x_i-\bar{x}{)}^2$, $y_i-\bar{y}$, $(x_i-\bar{x})(y_i-\bar{y})$, then $b_1$ as the ratio of the two column totals, then $b_0=\bar{y}-b_1\bar{x}$, and finally write the estimated equation $\hat{y}=b_0+b_{1}x$. You should also be able to predict $\hat{y}$ for a given $x$ by substitution. The exam mixes two kinds of questions: a few will be calculation-oriented, and a few will give an output and ask for your inference — interpretation skills matter right from the beginning, since you all know how to do the calculations.

7.5 Reading Regression Output: Simple and Multiple

7.5.1 The Three-Part Output

For the armored pizza trucking problem, the output (available in Excel) has three portions: one portion is the regression summary, another portion is the ANOVA, and the third portion is the regression coefficients. These are the three standard blocks of every regression output, in order:

1. Regression statistics — the summary block: the correlation coefficient $R$, the coefficient of determination $R^2$, the adjusted $R^2$, the standard error, and the number of observations. This block answers "how well does the model fit?"
2. ANOVA — the analysis-of-variance block: source (regression, residual/error, total), sum of squares, degrees of freedom, mean squares, the F statistic, and the F p-value. This block answers "is the model as a whole significant?"
3. Coefficients — the coefficients block: the intercept and each coefficient with its standard error, t statistics, and p-values for each predictor. This block answers "which variables contribute significantly?"

The column of interest for writing the equation is the coefficients column: the intercept is nothing but $b_0$ and the coefficient of $x$ is $b_1$ — the y-intercept is the constant, and the coefficient of x is the slope.

7.5.2 The Armored Pizza Trucking Example: The Equation from the Output

From the coefficients column of the simple regression output for the armored pizza trucking problem, the intercept is 1.27 and the slope is 0.067, so

$$\hat{y}=1.27+0.067x$$

The verbal description: "the way we wrote $b_0$ plus $b_{1}x$ where $b_0$ is 1.27 plus 0.067 on a similar note." If you are asked to write a regression equation from an output, the coefficients column gives you everything: $\hat{y}=b_0+b_{1}x$.

7.5.3 Overall Significance and Individual Significance

Two significance questions are answered by the same output. The ANOVA portion tells us the overall significance — whether the regression model as a whole is valid or not — captured by its p-value: if the p-value is less than alpha, say the standard $\alpha =0.05$, this minimum value indicates that the regression is significant, and it is a good model being suggested for this problem. The last column, the p-values of the coefficients, tells us about individual contributions: with the help of the p-value of each coefficient we can say whether each variable — for instance, the miles traveled — contributes significantly or not. For this particular example, looking at the p-values, both the overall model and the individual variable are significantly contributing.

The two questions are different and both get an answer from one output:

For the simple regression case the two questions usually agree, because with one predictor the overall F test and the t test on the slope are testing the same thing. The interesting separation appears when there are several predictors — the next subsection.

7.5.4 Multiple Linear Regression Output: A Preview

The same reading applies to multiple linear regression, discussed in more detail in the next session. There we write the model with two independent variables:

$$\hat{y}=b_0+b_1{x}_1+b_2{x}_2$$

For the multiple-regression output shown, the coefficients column lists the intercept and the two slopes, with the two slope values stated as 0.06 and 0.93. These numbers match the standard trucking example in the textbook: with miles traveled ($x_1$) and number of deliveries ($x_2$) as predictors of travel time, the estimated equation is

$$\hat{y}=2.869+0.061x_1+0.923x_2$$

so the professor's "0.06 and 0.93" are the rounded slopes for miles and deliveries respectively. Note that the slope for miles changed between the simple and multiple outputs (0.067 alone, 0.061 with deliveries added) — in multiple regression each coefficient is the effect of that variable with all other variables held constant, which is why it can differ from the simple-regression slope.

The interesting point to watch: at times the overall significance is there — the model is significant — but if you carefully observe the individual p-values, a few of the independent variables turn out not to be significantly contributing. That contrast between overall significance and individual significance is an important point to address as far as the regression concepts are concerned, and the discussion will continue from this output in the next session, for both simple and multiple linear regression.

7.5.5 Software for Regression

Real-world: in Excel, the option lives in the Data Analysis portion of the ribbon — you need to add the Analysis ToolPak add-in ("analysis tool pack"), which enables the Data Analysis menu, and then work out the regression there. For simple and multiple regression, Excel is enough. For logistic regression and the advanced regressions, there is no such option in the Data Analysis tool: you need to write R code, or you can get those results with Python. In SPSS, predefined options are also there, and logistic regression is also available in the add-ins/toolpak of SPSS. For simple and multiple regression the Excel route works fine.

7.5.6 Student Questions and Answers

Exam Guidance Summary

• Roughly 90% of everything discussed so far counts for the mid-semester exam. What has been covered up to now is what you can think of as the mid-semester material.
• ANOVA questions are tabulated. When an analysis of variance problem is asked, the expected format is the ANOVA table: source of variation, sum of squares, degrees of freedom, mean square, F statistic, p-value — the same picture Excel and Minitab print. Learn the arithmetic: $SST=SSTR+SSE$, degrees of freedom $K-1$, $n_T-K$, $n_T-1$, and $F=MSTR/MSE$ with MSE in the denominator.
• The decision rule is the p-value rule used throughout the course: $p\le \alpha$ → reject the null; $p>\alpha$ → accept the null. The F p-value is read from the right tail, exactly like the Z, t and chi-square p-values discussed earlier.
• Calculation and output-interpretation questions. A few exam questions will be calculation-oriented: given $x_i$ and $y_i$, compute $\bar{x}$, $\bar{y}$, the deviation columns, $b_1=\sum (x_i-\bar{x})(y_i-\bar{y})/\sum (x_i-\bar{x}{)}^2$, $b_0=\bar{y}-b_1\bar{x}$, and write the estimated regression equation. A few will give you an output and ask for your inference: write the regression equation from the coefficients column (intercept and slope), state the overall significance from the ANOVA p-value, and state the individual significance from the coefficient p-values.
• If Excel is used, the exam may be framed differently. If the class works with Excel for these problems, the mid-semester exam will be framed in a different fashion; the instructions on that will be given, so nothing needs to be decided in advance.
• Reference chapters: chapter 13 for design of experiments and analysis of variance, chapter 14 for regression. For further reading on design of experiments, the book Design of Experiments by Montgomery is a good choice.

The common thread in every exam item above: the exam rewards the table or output in front of you — the ANOVA table's six columns, the coefficients column's intercept and slope, the two p-values (ANOVA for the model, coefficients for the variables). Practicing the two worked examples of this session — the Chemtech ANOVA and the trucking regression — in the tabulated format covers both styles of question.

Key Industry Applications

• Manufacturing process selection: the Chemtech filtration system example chooses among three assembly methods (A, B, C) by comparing the number of units assembled per week. The one-way ANOVA tests whether the methods differ significantly, and the random assignment of workers to methods removes personal bias — the standard industrial experiment.
• Quality improvement through variance decomposition: ANOVA splits total variation into assignable causes (the factors you control) and chance causes (error). When a single factor accounts for 70% of the variation, 30% is left to chance; industry practice is to refine the design — add factors, use interactions — so the chance component shrinks. This is the same logic behind completely randomized, randomized block, and factorial designs used in agriculture, manufacturing, and product testing.
• Predictive modeling with regression: the trucking problem uses simple linear regression (Excel output $\hat{y}=1.27+0.067x$) to relate a predictor such as miles traveled to the response, and to forecast the response by substitution — the everyday use of regression for cost and demand prediction.
• Model validation in practice: the ANOVA p-value of a regression output validates the model as a whole, and the coefficient p-values validate each predictor individually. The warning case — an overall significant model with a non-significant variable — is a routine diagnostic in data science work, and the remedy is to eliminate the weak variable and remodel.
• Software practice: Excel's Data Analysis ToolPak handles simple and multiple linear regression; Minitab prints the full ANOVA table with exact p-values; logistic and advanced regressions require R code or Python; SPSS offers predefined options, with logistic regression in its add-ins.

Where this fits in the field: this session is the bridge from hypothesis testing (single means, variances) to the two workhorses of applied statistics — ANOVA for designed experiments in engineering and R&D, and regression for prediction in operations, marketing, and finance. Both share the same underlying move: decompose the variation of the response into explained and unexplained parts, and judge the explained part against the unexplained.