Logistic Regression Recap, Odds Ratios, and Principal Component Analysis

12.1 The Logistic Regression Equation and Why It Is Used

12.1.1 The Two Equivalent Forms of the Logistic Model

We open by revisiting logistic regression from the previous session, where it was introduced as an advanced regression technique in comparison with general linear regression and multiple linear regression. The logistic regression equation is usually written in one of two equivalent shapes. The first shape places the exponential in the numerator and the denominator:

$$p=\frac{e^{{\beta }_0+{\beta }_{1}x}}{1+e^{{\beta }_0+{\beta }_{1}x}}$$

The verbal description in class was "e power beta 0 plus beta 1x by 1 plus e power beta 0 plus beta 1x". The second shape is the same expression flipped into a one-over-one-plus-minus-exponential form: "1 by 1 plus e power minus beta 0 plus beta 1x":

$$p=\frac{1}{1+e^{-({\beta }_0+{\beta }_{1}x)}}$$

Both forms give the identical probability; the second one simply rewrites the exponent with a minus sign. Here is the algebra that shows why $\frac{e^a}{1+e^a}=\frac{1}{1+e^{-a}}$, with $a={\beta }_0+{\beta }_{1}x$:

$$\begin{array}{rlr}\frac{e^a}{1+e^a}& =\frac{e^a}{1+e^a}\cdot \frac{e^{-a}}{e^{-a}}& \text{(multiply numerator and denominator by }{e}^{-a}\text{)}\\ & =\frac{e^a\cdot e^{-a}}{(1+e^a)\cdot e^{-a}}& \text{(distribute }{e}^{-a}\text{)}\\ & =\frac{e^0}{e^{-a}+e^a{e}^{-a}}& \text{(combine the exponents: }{e}^a{e}^{-a}=e^0=1\text{)}\\ & =\frac{1}{e^{-a}+1}& \text{(the numerator is exactly 1)}\\ & =\frac{1}{1+e^{-a}}& \text{(same sum, terms written in the other order)}\end{array}$$

So the two shapes are the same model wearing two different outfits; whichever one appears in an output, the probability it produces is identical. Here $p=P(Y=1\mid x)$ is the expected value of the response given the predictor, ${\beta }_0$ is the intercept term, and ${\beta }_1$ is the coefficient on $x$. The symbol $e$ is Euler's number, $e\approx 2.71828$, the base of the natural logarithm. When the model has several predictors, the exponent simply grows more terms, ${\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2$, and the same structure holds:

$$p=\frac{e^{{\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2}}{1+e^{{\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2}}$$

12.1.2 Why an S-Shaped Curve for a 0/1 Response

The nature of $y$ drives the choice of model: the response must be either 0 or 1, that is, it must be a categorical or classified variable. Whenever the dependent variable behaves like this — a variable built from 0s and 1s — the best model to recommend is this S-shaped curve, the logistic function, and that is exactly where the exponential term shows up. A straight line is the wrong tool for a 0/1 outcome; the S-shape keeps the predicted values trapped between 0 and 1.

The example below runs the formula at four values of $x$ to see that saturation happen with real numbers.

To see the shape, picture the graph with the independent variable $x$ on the horizontal axis and the probability $p$ on the vertical axis (from 0 to 1). The curve is S-shaped: it starts nearly flat at the bottom left, hugging 0, then rises steeply through the middle, then flattens again at the top right, approaching 1 without ever reaching it. The midpoint of the climb happens where the exponent equals 0, which here means ${\beta }_0+{\beta }_{1}x=0$, so $x=-{\beta }_0/{\beta }_1=7/3\approx 2.33$: at that point $p=e^0/(1+e^0)=1/2=0.5$. Two landmarks matter: the curve never crosses the 0 line and never crosses the 1 line (both are asymptotes), and the steepest rise is near the middle. The takeaway: one formula produces every probability from near-0 to near-1, with the steep part reserved for values of $x$ near the midpoint.

12.1.3 Estimating the Coefficients from the Output

The second important point is how ${\beta }_0$ and ${\beta }_1$ are estimated; these are the regression coefficients, or logistic regression coefficients. In simple linear regression, partial derivatives give us ${\beta }_0$ and ${\beta }_1$ directly: we take the derivatives of the error sum of squares, equate them to 0, and solve for closed-form formulas for both coefficients. Logistic regression behaves nonlinearly, so this closed-form route is beyond the scope of the course. To dodge that mathematical difficulty, we estimate ${\beta }_0$ and ${\beta }_1$ with the help of the output: we read the coefficient estimates straight from the software output and anticipate their values by looking at the numbers there. This description covers the one-predictor structure, ${\beta }_0+{\beta }_{1}x$; with more predictors we simply extract each coefficient the same way. In practice the software reaches these numbers by an iterative numerical algorithm (the one R labels "Fisher's scoring", seen in Section 12.7), but the exam skill is reading the finished estimates, not re-running the algorithm by hand.

There is also an interpretation warning attached to these coefficients. It is not the straightforward linear-regression reading — "one unit of increase in $X_1$, what is the influence on $Y$?" That reading does not carry over cleanly, because the entire expected value of $Y$ is treated as a probability. The measure we use to interpret the model instead is the odds ratio, covered in Section 12.2.

In practice, this is how binary-outcome models are read in business and health analytics: a bank estimating the probability that a credit-card application is approved, a retailer estimating the chance that a mailed coupon is redeemed, a doctor estimating the chance of a disease given test results. Logistic regression is the standard first tool for such yes/no predictions, and its output is interpreted through odds and the odds ratio, the next stop on the arc.

12.1.4 Student Questions and Answers

That settles the opening question. The takeaway below is the one to carry into the next section.

12.2 Odds and the Odds Ratio

12.2.1 Odds: Success over Failure

The odds ratio is the very important measure we use to interpret logistic regression. Many software packages print the formula or the value directly in the output; the only real skill on our side is interpreting whatever numerical value we get. The odds of an event are defined as the probability of the success divided by the probability of not having the success:

$$\text{Odds}=\frac{p}{1-p}$$

where $p$ is the probability of the event we code as 1. If $p=0.5$, the odds are 1, meaning success and failure are equally likely; as $p$ climbs toward 1, the odds grow without bound.

A table makes the difference between the two scales visible, with the probability on one side and the equivalent odds on the other:

The two numbers carry the same information, but they live on different scales: probability is trapped between 0 and 1, odds runs from 0 to infinity. Pick the odds when you want multiplicative comparisons ("three times as likely in odds"), and stay with probability when you want an answer between 0 and 1.

12.2.2 The Odds Ratio from the Coefficients

For logistic regression there is a simple relationship that lets us anticipate the odds ratio from the coefficient column of the output. To find the odds ratio for $X_1$, we raise $e$ to the power ${\beta }_1$; for $X_2$, we raise $e$ to the power ${\beta }_2$:

$$\text{Odds ratio for }{X}_k=e^{{\beta }_k}$$

The ${\beta }_k$ values are exactly the numbers in the coefficients column of the output. This is the shortcut we use to anticipate the odds ratio without any further computation.

Reading the number: an odds ratio of 1 means the two events have identical odds, so the variable changes nothing. An odds ratio greater than 1 means the predictor raises the odds of the event — for example 3 means "three times the odds". An odds ratio below 1 means the predictor lowers the odds — for example 0.5 means "half the odds".

12.2.3 Where the Numbers Live: Minitab versus R

There is a practical difference between the two software outputs used in class. In the Minitab output the odds ratio is already present, printed in the output itself. In the R output the odds ratio is not printed for you, so you must explicitly calculate it with $e^{{\beta }_k}$, taking ${\beta }_k$ from the coefficients column and then interpreting that number.

Real-world: this is how working analysts read logistic regression output — Minitab hands you the odds ratio ready-made, while R users compute $e^{\beta }$ from the coefficients column as a routine step. The same habit shows up across statistics: medical researchers report odds ratios for risk factors, marketers report odds of redemption per customer segment, and credit teams report how much a predictor multiplies the odds of default.

12.3 Worked Example: Simon Store with Two Predictors

12.3.1 The Data and the Estimated Equation

We revisit the Simon Store example introduced in the previous session, where the data has a response $Y$ and two independent variables $X_1$ and $X_2$. In this example $Y$ records whether the customer used a coupon or not (coded 1 for using the coupon, 0 for not using it), $X_1$ is the annual spending of the customer (a continuous variable), and $X_2$ indicates whether the customer holds a Simon credit card (coded 1 for yes, 0 for no). One variable is continuous and the other is categorical, both feeding the single 0/1 response. A pilot study on a small amount of data was worked in the previous session, and we now carry the same problem further.

The study behind the numbers: Simon sent an expensive four-color catalog with a 50-dollar discount coupon to a random sample of 100 customers — 50 cardholders and 50 non-cardholders — and recorded who used the coupon. The first ten rows of the data set look like this (spending in thousands of dollars, so 2.291 means 2,291 dollars):

From the output we write the estimated logistic regression equation, taking spending as $x_1$ and credit-card holding as $x_2$. The intercept was written as 2.14 only because of the space constraint on the slide — the coefficient was truncated to two digits — and the coefficients shown are 0.34 for $x_1$ and 1.09 for $x_2$:

$$\hat{p}=P(Y=1\mid x_1,x_2)=\frac{e^{-2.14+0.34x_1+1.09x_2}}{1+e^{-2.14+0.34x_1+1.09x_2}}$$

Two clarifications pin the equation down exactly. First, the minus sign belongs to the intercept: the intercept estimate is negative, $-2.14$ as printed (full precision $-2.146$), while both coefficients are positive, 0.34 and 1.09 (full precision 0.342 and 1.099). Second, the two equivalent shapes of the logistic model (Section 12.1) were mixed while reading the equation aloud — "e power minus 2.14 plus 0.34x1 plus 1.09x2" — but the shape written above is the working one, since it reproduces the fitted probabilities in the output. As a check: at $x_1=2$, $x_2=1$, the exponent is $-2.14+0.34(2)+1.09(1)=-0.37$, giving $e^{-0.37}/(1+e^{-0.37})\approx 0.41$, exactly the table value 0.4102; with the full-precision coefficients $-2.146+0.684+1.099=-0.363$, the result is 0.4102 to four digits.

Real-world: this exact workflow — a retailer mailing a costly catalog only to customers whose fitted probability of redemption is high — is how direct-mail targeting runs at scale. The fitted probabilities themselves drive the mailing decision (send only above a 0.40 cutoff), while the odds ratios explain to management why the card variable matters.

12.3.2 Worked Computation: Odds for x1 = 2 and x2 = 1

The output table gives the fitted probability for each combination of the predictors. For a customer who spent 2,000 dollars last year ($x_1=2$, spending measured in thousands) and has a Simon credit card ($x_2=1$), the table value is:

$$P(Y=1\mid x_1=2,x_2=1)=0.4102$$

The odds in favor of using the coupon for these customers are then:

$$\begin{array}{rlr}\text{Odds}& =\frac{0.4102}{1-0.4102}& \text{(odds = success over failure)}\\ & =\frac{0.4102}{0.5898}& \text{(failure probability is }1-0.4102\text{)}\\ & \approx 0.6956& \end{array}$$

So we take the probability of success, 0.4102, divide by the probability of failure, $1-0.4102=0.5898$, and land at 0.6956. The odds of using the coupon for a 2,000-dollar spender with a Simon card are about 0.70.

12.3.3 Worked Computation: Odds for x1 = 2 and x2 = 0

Now take the same spending level but a customer without a Simon credit card, $x_2=0$. The table gives:

$$P(Y=1\mid x_1=2,x_2=0)=0.1881$$

Only this single value changes; everything else in the calculation is the same pattern. The odds in favor of using the coupon here are:

$$\begin{array}{rlr}\text{Odds}& =\frac{0.1881}{1-0.1881}& \text{(odds = success over failure)}\\ & =\frac{0.1881}{0.8119}& \text{(failure probability is }1-0.1881\text{)}\\ & \approx 0.2318& \end{array}$$

12.3.4 Worked Computation: The Odds Ratio

The odds ratio is nothing but the odds with $x_2=1$ divided by the odds with $x_2=0$:

$$\begin{array}{rlr}\text{Odds ratio}& =\frac{\text{Odds}(x_2=1)}{\text{Odds}(x_2=0)}& \text{(odds for cardholders over odds for non-cardholders)}\\ & =\frac{0.6956}{0.2318}& \text{(the two odds just computed)}\\ & \approx 3.0& \end{array}$$

12.3.5 The Invariance Property of the Odds Ratio

There is a deeper point hiding in this computation: the odds ratio for each independent variable is computed while holding all other independent variables constant — but it does not matter which constant value you hold them at. We did the division with $x_1=2$ and $x_2=1$ versus 0 and got about 3. If instead we set $x_1=3$, $x_1=4$, or $x_1=5$ (spending of 3,000, 4,000, or 5,000 dollars) and again compare $x_2=1$ against $x_2=0$, the odds ratio still comes out somewhere around 3. It is immaterial what value you take for $x_1$; keeping the other variables constant at any level leaves the odds ratio unchanged.

Check it with the output's fitted probabilities for three different spending levels:

The algebra shows why the constant never matters. The odds at any point equal the exponential in the model, $\text{Odds}=e^{{\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2}$ (Section 12.2), so the odds ratio for $x_2$ is:

$$\frac{e^{{\beta }_0+{\beta }_1{x}_1+{\beta }_2}}{e^{{\beta }_0+{\beta }_1{x}_1}}=e^{{\beta }_2}$$

The terms containing $x_1$ cancel out of the ratio completely — whatever level of spending you hold constant, the result is the same single number $e^{{\beta }_2}$.

That is the beauty of the odds ratio and the reason we reach for it when interpreting logistic regression: it gives one stable, comparable number per predictor instead of a reading that shifts with the point on the curve. Whenever you talk about the odds ratio for $x_2$, you are keeping $x_1$ at some constant; the constant itself is irrelevant to the ratio.

The whole worked example, in one breath:

12.4 Worked Example: Simon Store with One Predictor

12.4.1 The Estimated Equation from the R Output

To show what changes when a predictor is dropped, the annual spending column was deleted entirely from the Simon Store data and the R code was run again with only the credit-card variable left. The expected output, exactly what R produces, is the basis for the whole exercise. Asked to write the logistic regression equation relating $x$ and $y$, we write the estimated equation:

$$\hat{p}=\frac{e^{-0.94+1.02x}}{1+e^{-0.94+1.02x}}$$

where $x$ is the Simon credit-card indicator (1 for having the card, 0 for not having it). The coefficient on $x$ in the output is $b_1=1.0245$, which is where the 1.02 in the equation comes from: the equation is written with the rounded value 1.02, and every computation in this session uses 1.02 (the coefficient column value 1.0245 rounds to 1.02; a single spoken pass said "1.04x", a slip of the tongue — the printed equation carries 1.02). The intercept estimate is $-0.94$.

With one predictor, the write-versus-estimate distinction is worth keeping straight: to write the equation we use the symbolic form with ${\beta }_0$ and ${\beta }_1$,

$$p=\frac{e^{{\beta }_0+{\beta }_{1}x}}{1+e^{{\beta }_0+{\beta }_{1}x}}$$

and to estimate it we replace those symbols with the values in the output: ${\beta }_0$ becomes $-0.94$ and ${\beta }_1$ becomes 1.02. This two-step habit — symbolic form first, numbers second — is the same drill used for every logistic output in the course.

12.4.2 Worked Computation: Odds Ratio and Its Interpretation

The only predictor here is the credit-card indicator, so there is one odds ratio to compute. Using the shortcut formula:

$$\text{Odds ratio for }x=e^{b_1}=e^{1.0245}\approx 2.78$$

You can verify this with a calculator: $e$ to the power 1.025 lands at 2.78 (more precisely $e^{1.0245}=2.7859$, and the rounded coefficient gives $e^{1.02}=2.7732$ — either way, about 2.78).

A comparison of the two fitted models shows what one variable's presence changes:

When to pick which: use the two-predictor model when you want the card effect with spending controlled; use the one-predictor model when spending data is unavailable — and remember the odds ratio then mixes the card effect with the spending differences between cardholders and non-cardholders.

The one-predictor example closes with its summary:

12.5 The G Statistic and Model Significance

12.5.1 Null Deviance minus Residual Deviance

Model significance for logistic regression is checked with the G statistic, which is nothing but the chi-square statistic. The formula from R is simply the difference between the null deviance and the residual deviance:

$$G=\text{null deviance}-\text{residual deviance}$$

For the single-predictor Simon Store example:

$$\begin{array}{rlr}G& =\text{null deviance}-\text{residual deviance}& \\ & =134.60-128.53& \text{(the two deviance values in the R output)}\\ & =6.07& \end{array}$$

with degrees of freedom 1. The degrees of freedom are not computed here; they come with the output, and for this model it is 1.

12.5.2 The Decision Rule

The G statistic tests the model as a whole. The hypotheses are the same in spirit as in multiple regression:

• $H_0$: all coefficients on the predictors are 0 — the model is no better than the intercept-only model.
• $H_a$: at least one coefficient is not 0 — the predictors add real explanatory power.

To conclude at a chosen level of significance whether a significant difference exists, we consult the chi-square distribution table at 1 degree of freedom and validate the value against it. In Minitab, the p-value corresponding to the chi-square value is printed in the output, and we can conclude directly from the p-value. If the p-value is not printed, it is our responsibility to calculate it explicitly and compare. The logic is the same as always: if the p-value is less than $\alpha$, reject the null hypothesis; otherwise accept the null hypothesis.

Real-world: this is the significance gate that runs before any logistic result is reported — journals, regulators, and model-review boards all want to know whether the fitted model beats the intercept-only baseline, and the printed p-value next to the chi-square is the standard evidence. The G statistic does for logistic regression the same job that $R^2$ and adjusted $R^2$ do in linear regression output: one number that summarizes whether the model is worth taking seriously.

12.5.3 Student Questions and Answers

The decision rule, distilled:

12.6 The Logit Function: When the Log of Odds Is Linear

12.6.1 Deriving the Relationship between Odds and the Model

There is an interesting relationship between the odds and the exponent in the logistic regression equation, and it is worth deriving once in full. We know the simple formula for odds is $p$ by $1-p$, and the logistic regression equation gives $p=\frac{e^{{\beta }_0+{\beta }_{1}x}}{1+e^{{\beta }_0+{\beta }_{1}x}}$. If this is $p$, then $1-p$ is $\frac{1}{1+e^{{\beta }_0+{\beta }_{1}x}}$, and we substitute both into the odds:

$$\frac{p}{1-p}=\frac{\frac{e^{{\beta }_0+{\beta }_{1}x}}{1+e^{{\beta }_0+{\beta }_{1}x}}}{\frac{1}{1+e^{{\beta }_0+{\beta }_{1}x}}}$$

The denominator of the compound fraction flips and the common term $1+e^{{\beta }_0+{\beta }_{1}x}$ cancels between numerator and denominator:

$$\begin{array}{rlr}\frac{p}{1-p}& =\frac{\frac{e^{{\beta }_0+{\beta }_{1}x}}{1+e^{{\beta }_0+{\beta }_{1}x}}}{\frac{1}{1+e^{{\beta }_0+{\beta }_{1}x}}}& \text{(substitute }p\text{ and }1-p\text{ into the odds)}\\ & =\frac{e^{{\beta }_0+{\beta }_{1}x}}{1+e^{{\beta }_0+{\beta }_{1}x}}\cdot \frac{1+e^{{\beta }_0+{\beta }_{1}x}}{1}& \text{(dividing by a fraction multiplies by its reciprocal)}\\ & =e^{{\beta }_0+{\beta }_{1}x}& \text{(the common denominator cancels out)}\end{array}$$

So the odds are exactly the exponential in the logistic equation. Next, take natural logs on both sides; the $\ln$ and the $e$ cancel:

$$\begin{array}{rlr}\ln \left(\frac{p}{1-p}\right)& =\ln \left(e^{{\beta }_0+{\beta }_{1}x}\right)& \text{(apply ln to both sides)}\\ & ={\beta }_0+{\beta }_{1}x& \text{(ln and }e\text{ undo each other)}\end{array}$$

This is the step that deserves emphasis: the natural log of the odds in favor of $Y=1$ is a linear function of the independent variable. What started as a nonlinear S-curve becomes a straight line once we take the log of the odds.

12.6.2 The Logit Function

That linearized quantity has a name: the logit function. We write it as

$$g(x)=\ln \left(\frac{p}{1-p}\right)={\beta }_0+{\beta }_{1}x$$

and its estimate as $\hat{g}(x)=b_0+b_{1}x$, using the fitted coefficients $b_0$ and $b_1$ from the output. So the model is linear in the log-odds space even though it is nonlinear in probability space.

This is also why log transformations are used in practice: to convert nonlinear relationships into linear ones and then work out the entire analysis in the linear space. From the logistic equation we get two observations worth keeping: first, what odds are and how to interpret the odds ratio; second, the natural log of odds always equals a linear function of the independent variable — that is what the logit transformation is all about.

Real-world: the logit link is the standard bridge in applied work — analysts fit a linear model to log-odds and convert predictions back to probabilities with the S-curve. Banks, insurers, and medical researchers all report their logistic findings in logit form, because the linear coefficients are easy to compute, to test, and to explain in the log-odds space.

12.6.3 Worked Computation: Estimated Probabilities from the Estimated Logit

For the single-predictor Simon Store example, the estimated logit is $\hat{g}(x)=-0.94+1.02x$, straight from the estimated equation. To recover the estimated probability of using the coupon, we invert the logit:

$$\hat{y}=\frac{e^{\hat{g}(x)}}{1+e^{\hat{g}(x)}}$$

For a customer who does not have a Simon credit card, substitute $x=0$:

$$\begin{array}{rlr}\hat{y}& =\frac{e^{-0.94}}{1+e^{-0.94}}& \text{(the logit at }x=0\text{ is }-0.94)\\ & =\frac{0.3906}{1.3906}& \text{( }{e}^{-0.94}\approx 0.3906\text{)}\\ & \approx 0.28& \end{array}$$

Try that on a calculator: the exponential of minus 0.94 over 1 plus the exponential of minus 0.94 gives about 0.28 (the full-precision output value quoted in the session is 0.279 — the tiny difference is the rounding of the printed coefficients). For a customer who has the Simon credit card, replace $x$ with 1:

$$\begin{array}{rlr}\hat{y}& =\frac{e^{-0.94+1.02}}{1+e^{-0.94+1.02}}& \text{(the logit at }x=1\text{ is }0.08)\\ & =\frac{e^{0.08}}{1+e^{0.08}}& \text{( }{e}^{0.08}\approx 1.0833\text{)}\\ & \approx 0.52& \end{array}$$

12.6.4 Practice Problem: Textbook Page 737, Problem 46

There is another simple example to practice the whole drill: the textbook problem TXMA page 737 problem 46, with one dependent and one independent variable, where the data was collected, the problem worked through, and the output already given. The setting: a bank wants more customers to sign up for payroll direct deposit, and believes the average monthly checking balance predicts who will sign up. The dependent variable $y$ is coded 1 if the customer signed up for payroll direct deposit and 0 if not; the independent variable $x$ is the average monthly checking balance (in hundreds of dollars).

Work through it in this order: (1) write the logistic regression equation relating $x$ and $y$, expressed in terms of ${\beta }_0$ and ${\beta }_1$; (2) estimate the equation by replacing ${\beta }_0$ and ${\beta }_1$ with the values in the output; (3) compute the estimated logit, $e^{b_0+b_{1}x}$ as the linear combination; (4) compute the estimated probability; (5) find the odds ratio for $x_1$ and state the conclusion by looking at that single number.

12.7 Reading the R Output: AIC, Fisher's Scoring, and Significance Stars

12.7.1 AIC and Model Comparison

The R output carries more values than we used so far, and two of them deserve a brief understanding. The first is the AIC value. AIC is the Akaike Information Criterion — a full form worth remembering. Generally speaking, this value is useful when we compare different models against each other. For an individual model standing alone, there is no good AIC value and no bad AIC value; deciding anything from a single number does not matter. When we do have a comparison among models, AIC takes care of that point: the model with the lowest AIC offers the best fit in terms of the AIC value — if it is lower and lower, it is the better fit. We simply cannot justify anything by looking at one value in isolation. BIC and many other criteria of the same family exist; in time series models, all these metrics are routinely used to justify a choice.

Real-world: AIC and BIC are standard tools in model selection, especially in time series work, where candidate models are ranked by these criteria and the lowest wins. The same habit shows up whenever a statistician has to choose among competing specifications — the criteria never judge one model alone, they rank a field of candidates.

12.7.2 Fisher's Scoring and Convergence

The second value is Fisher's scoring, which talks about the convergence of the model. There are two states: converging and diverging. When the output shows "Fisher's scoring needed six iterations to perform the fit", that is the typical count — in general, six iterations are required, and if the count comes in below that, it is well and good. It is a numerical algorithm of sorts, reporting how the model converged on the available data set. For huge data sets, these scores help us talk about convergence behavior.

Real-world: Fisher's scoring is the iterative numerical procedure statistical software runs to fit logistic-type models; the iteration count is the convergence report that practitioners glance at when fitting on large data. On very large data sets, analysts watch this line because a model that takes hundreds of iterations — or never converges — usually signals a data problem, such as a predictor that separates the two outcome groups perfectly.

12.7.3 Significance Stars and p-values

The significance codes — the three stars, two stars, and so on — are the interpretation with the help of p-values. More stars means more significant: three stars appear when the p-value is much, much smaller. The rule stays the same: compare $p$ with $\alpha$. This connects to the previous session's discussion of how to justify model significance in various ways — $R^2$, adjusted $R^2$, and the other values in different outputs. In logistic regression, the chi-square value (the G statistic) tells us, to some extent, the significance of the model.

12.7.4 Student Questions and Answers

From the output reading, one summary line:

12.8 Dimensionality Reduction: The Big Idea

12.8.1 Definition and Objective

We now switch to a new topic: reducing the dimension of a data set. If the original data is, say, five-dimensional, we may want to move it to three-dimensional, two-dimensional, or one-dimensional space; that transformed data is what the reduction produces. Dimensionality reduction is the transformation of data from a higher dimensional space into a lower dimensional space that retains the meaningful properties of the original data. The phrase to underline is "retains": we should maintain the maximum amount of information present in the original data — not at the cost of that information. Another way to say it: we reduce the size of the data set while keeping as much of the variation as possible. Reducing from higher to lower dimension while preserving this objective is the whole game.

12.8.2 Variation Is Information

A compact identity drives everything that follows: variation, in the statistical sense, is information. Information, in the statistical sense, is variation — the two go together. The amount of information in a data set is nothing but the amount of variation, the maximum variation the set is going to capture. So when we ask how well a reduction performs, we ask how much of the original variation it keeps; keeping the maximum amount of variation is the objective of every reduction technique.

Put the identity to work as a working rule:

12.8.3 Two Main Techniques: PCA and LDA

Two reduction techniques stand out: Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA). People working in machine learning and AI call these unsupervised and supervised learning, and the terms are used there all the time; from a mathematical perspective, as a mathematical teacher would put it, these words are not used that frequently — the techniques are mathematical techniques. Anyone with a bit of linear algebra, who knows eigenvalues and eigenvectors, will find the conceptual understanding of PCA and LDA easy, because those tools are exactly what PCA and LDA rest on.

The name of PCA already indicates the idea: you transform your original data into its transformed data, and that transformed data is what we call the principal components. It is another way of thinking, another way of representing the same data set, and the question is how to find those principal components while achieving the objective of maximum retained variation. LDA works the other way: it relates to classification — the same kind of problem as logistic regression, where the response is 0s and 1s. LDA also reduces dimension, but with the help of a line it classifies the two different classes, and there should be a clear separation between those classes. The mathematics of that separation is what LDA provides. In this session we focus on the conceptual understanding of PCA and its numerical illustration, and we keep the math light — just what is relevant and digestible.

When to pick which: use PCA when the task is compression or visualization of unlabeled data; use LDA when the real task is separating labeled classes and you also want the dimension reduced.

Real-world: in machine learning and AI pipelines, PCA and LDA are the workhorses for shrinking huge feature sets before modeling, known there as unsupervised and supervised dimensionality reduction. In research practice, factor-style methods built on the same linear algebra are used to summarize dozens of survey questions into a few underlying dimensions.

12.8.4 A Bit of History: Hotelling, 1933

PCA is a simple data reduction technique with a long pedigree: way back in 1933, the famous scientist Hotelling pioneered the concept behind principal component analysis. A related test, the Hotelling statistic, is also available in the literature, so the name comes up in more than one place. When you meet "Hotelling" again in multivariate statistics, it is the same lineage: the statistic is named after the same Harold Hotelling whose 1933 paper introduced what we now call principal components.

12.8.5 The Two Perspectives of PCA

PCA is always discussed with two perspectives in view. The first is the lower dimension — the main objective we want to fulfill. The second is the orthogonality of the new dimensions — what we require of the new axes. Both perspectives must be maintained whenever we reduce from a higher dimension to a lower one. What orthogonality means exactly is developed later; for now the two perspectives frame the technique: PCA must (1) land in a lower-dimensional space and (2) do it with new axes that are perpendicular to each other, so that none of the new dimensions repeats information carried by another.

12.8.6 Student Questions and Answers

A few traps to avoid while carrying the big idea forward:

The big idea, wrapped up:

12.9 Principal Component Analysis: The Geometric Idea

12.9.1 The Data Setup: n Observations on Two Variables

Consider a simple two-dimensional data plot: a data set with $n$ observations and two variables, or two input features. With 15 observations and two attributes we would write the first column as $x_{11},x_{21},x_{31},\dots ,x_{n1}$ and the second as $x_{12},x_{22},x_{32},\dots ,x_{n2}$. The rows are the observations and the columns are called the attributes; the whole data set is an $n\times 2$ matrix, and a data set laid out in rows and columns is easily handled with matrix algebra — those who know a bit of matrices find this easy.

Plot the same observations as a scatter plot with $x_1$ on one axis and $x_2$ on the other, and inspect the points. The data shows a positive relationship; a strong linear relationship is visibly present among the two variables. To justify that numerically we build a correlation matrix.

12.9.2 The Correlation Matrix

The correlation matrix for the two variables has 1s on the diagonal — the correlation of $x_1$ with itself and of $x_2$ with itself is always 1 — and the off-diagonal entries $r_{12}$ and $r_{21}$ hold the correlation between variable 1 and variable 2; since correlation between 1 and 2 is one and the same as correlation between 2 and 1, both positions carry the same value:

$$\mathbf{R}=\left(\begin{array}{cc}1& r_{12}\\ r_{21}& 1\end{array}\right)=\left(\begin{array}{cc}1& r_{12}\\ r_{12}& 1\end{array}\right)$$

Correlation always lies between $-1$ and $+1$. We do not know the actual observations here, but common sense from the pattern tells us $r_{12}$ must be very close to 1 — maybe 0.9 or 0.8. If we explicitly calculate the correlation coefficient we should anticipate a huge value: a strong linear relationship is present, and we can convince ourselves that $r_{12}$ is around 0.9 given the behavior we witness.

12.9.3 The Covariance Matrix

To capture the amount of variation along $x_1$ and the amount of variation along $x_2$, we use the covariance matrix. For the two-variable case it looks like:

$$\mathbf{S}=\left(\begin{array}{cc}{S}_{11}& S_{12}\\ S_{21}& S_{22}\end{array}\right)$$

The diagonal entries are the variances: $S_{11}$ is the variation of $x_1$ and $S_{22}$ is the variation of $x_2$. The off-diagonal entries $S_{12}=S_{21}$ are the covariance between $x_1$ and $x_2$. If the population variance ${\sigma }^2$ is unknown — the usual situation — we replace it with the sample variance $s^2$, provided the sample is enough to generalize to the population.

Inspect the diagonals. $S_{11}$ and $S_{22}$ are not the same, but both capture a substantial amount of variation; we cannot ignore either $x_1$ or $x_2$, because dropping either one would throw away a substantial share of the variation, and with it a share of the information, in the original data. In the original axes there is no such interesting observation yet — both variables matter roughly equally. (For the worked example above: $S_{11}=2.5$, $S_{22}=4.3$, and $S_{12}=S_{21}=3.0$, so the matrix is symmetric and both diagonals are substantial.)

12.9.4 Rotating the Axes: New Dimensions Z1 and Z2

Now think about the same problem differently: rotate the $x_1$-axis and the $x_2$-axis rigidly, anticlockwise, by an angle $\theta$. As we rotate, we form new axes $Z_1$ and $Z_2$ — the new dimensions for the data set. Those new dimensions are the transformed data, and they are what we call the principal components. The transformation involves the rotation angle; written for one of them:

$$Z_1=x_1\cos \theta +x_2\sin \theta$$

Apply the same rotation to every observation of the worked example and the variation moves. The five points $(1,1),(2,1),(3,2),(4,3),(5,6)$ become $Z_1=(1.41,2.12,3.54,4.95,7.78)$ and $Z_2=(0,-0.71,-0.71,-0.71,0.71)$: the sample variance along $Z_1$ is 6.4 and along $Z_2$ is 0.4. In the original data the variation along $x_1$ and along $x_2$ was comparable (2.5 and 4.3). After rotation the variation along $Z_1$ is much larger than the variation along $Z_2$ — the cloud stretches along $Z_1$ and squashes along $Z_2$. Along the $Z_1$ direction there is much more information; along the $Z_2$ direction there is little.

12.9.5 Reducing Two Dimensions to One

Because information is variation, the comparison of variances decides the reduction. The variation of $Z_1$ is greater than or equal to the variation of $Z_2$, so we retain $Z_1$ and ignore $Z_2$ for further discussion. That is the whole conceptual understanding of the transformation: the same original data, expressed in rotated coordinates, concentrates its variation into the first new axis. For the two-dimensional problem, we compare the variation along $Z_1$ with the variation along $Z_2$ and conclude whether a single dimension suffices; if it does, that single dimension is the first principal component. In the worked example the single coordinate $Z_1$ alone carries 6.4 of the total variation of 6.8 — about 94% — which is the quantitative meaning of "the reduction retains the meaningful properties".

12.9.6 The General Case: From P to M Dimensions

The same logic scales. If there are $P$ independent variables, or features, we try to reduce them to $M$, and always $M\le P$. The components $Z_1,Z_2,\dots$ are ordered by captured variation: $Z_1$ takes the maximum amount of variation from the original data, $Z_2$ takes the next best share, $Z_3$ the next, and so on. With 10 features, one component may capture most of the variation, and the question becomes whether 2 components are enough or whether more are needed, and how to decide. The mathematics behind all this — how to calculate these variances, how to decide which component contributes more — is handled with the covariance matrix and then eigenvalues and eigenvectors, which are the elementary linear algebra tools we need to strengthen this discussion. The full numerical procedure is worked next session.

12.9.7 The Ellipse Intuition: Correlated to Uncorrelated

There is a clean geometric way to see what PCA does. The original data is correlated data; the transformed data is uncorrelated. Think of the data cloud as an ellipse with a major axis and a minor axis. If the major and minor axes do not coincide with the $x_1$ and $x_2$ axes, the variables are correlated with each other. When the major and minor axes coincide with $x_1$ and $x_2$, the data is uncorrelated. PCA rotates the coordinate system so that the new axes line up with the ellipse's own axes, which turns correlated variables into uncorrelated ones.

One question is left hanging on purpose: why do we need uncorrelated data — what is the reason behind transforming a correlated data set into an uncorrelated one? Think about it; the reason is developed along with the mathematics next session.

12.9.8 Where the Mathematics Goes

The projections themselves are not worked in detail yet — the $Z_1$ and $Z_2$ transformations carry trigonometric terms because of the rotation by $\theta$. The maximum contribution, or the maximum amount of information, and which direction is enough, are all computed with the help of the covariance matrix, eigenvalues, and eigenvectors. That is the math agenda for the next session.

12.10 Linear Discriminant Analysis: A First Look

12.10.1 When PCA Is Not Enough

Consider a simple two-dimensional plot of height and weight: $x$ is height, $y$ is weight, with a few observations. The idea of PCA up to now gives us one motto: take the given two-dimensional plot and reduce it to a single-dimensional plot, projecting all the data points onto one dimension. With some knowledge of PCA we reduce the two-dimensional cloud into one dimension and project the observations onto it. Reducing the dimension is fine, but for this data set the real problem is a classification problem: how best to classify the different groups of points? Looked at closely, the data is not separable — there is no single line that separates the two classes distinctly. This exposes the boundary of PCA: PCA is not a classification technique. It reduces dimensions; it does not assign points to classes.

The picture to hold on to: a scatter plot with height on the horizontal axis and weight on the vertical axis, and two clouds of points (for example, customers who buy a product versus customers who do not). The two clouds overlap — a tall light person and a short heavy person can sit side by side — so no straight line cleanly cuts the plane into a buyers' side and a non-buyers' side. PCA would happily compress this plane into one axis, but the compressed axis mixes the two groups, because PCA never looked at the group labels in the first place.

Real-world: projecting data onto one or two principal components is common practice for visualizing and compressing large data sets, but classification requires a supervised step on top. This is why real pipelines pair a reduction step with a classifier rather than expecting the reduction to do the labeling.

12.10.2 LDA: The Line That Separates the Classes

To separate the two classes we introduce another interesting technique: linear discriminant analysis. The problem is the one anticipated above — the classes are not separable by an obvious line — and LDA is built exactly for it. With the help of a line, LDA classifies the two different classes; there should be a clear separation among them, and the question is which line best classifies the two classes, and what the mathematics behind that line is. LDA relates naturally to everything done so far: it is a classification problem with 0s and 1s, the same territory as logistic regression, but approached through dimension reduction and separation.

A compact view of where each tool in this session belongs:

When to pick which: fit logistic regression when you want probabilities and odds ratios for a binary outcome; use PCA when you have unlabeled data and need fewer dimensions; call on LDA when the classes are labeled, the data is high-dimensional, and both separation and reduction are wanted.

12.10.3 What Comes Next

Two things are coming up next session. First, a numerical illustration of PCA with a simple two-dimensional data plot, worked fully to get complete understanding — the same logic extends to huge-dimensional data in software, and you can reproduce the results in any software on your own problems. Second, the introduction of LDA itself. Both draw on a bit of linear algebra: matrix multiplication, how to calculate eigenvalues and their roots, and so on. The material is not in the regular text; it was assembled from different sources and will be shared.

Exam Guidance Summary

The administrative and revision-relevant points from this session, gathered in one place:

• Group assignment — 15 marks. Do not neglect or ignore that 15 marks. Submit a single report per team of about eight to nine pages, relevant to the case study, with all the ID numbers of the group members on the first page, submitted where instructed. One data set is provided per team. Adding interesting interpretations beyond the questions posed by the case study is advantageous — it helps you learn and is interesting for us to learn.
• Assignment logistics. Detailed instructions and the data sets will be uploaded by the 24th. The data sets are simple and come from the text itself.
• Quizzes. The quizzes carry 5 marks each — 5 for quiz 1 and 5 for quiz 2.
• Practice problem. TXMA page 737 problem 46: write the logistic regression equation relating $x$ and $y$, estimate it, compute the estimated logit, compute the estimated probability, find the odds ratio for $x_1$, and conclude. The output is already given; go through it and build a working understanding of how to interpret these values and estimate the probabilities.
• Resources. The Simon Store data and the bank data are shared for practice. The PCA/LDA material is not from the regular text; it will be shared from the sources used.
• Next session. The numerical illustration of PCA and the introduction of LDA, with eigenvalues and eigenvectors, come in the next session (29th).

Two more administrative questions came up about the report format and the marks:

And the marks breakdown was asked one more time:

Key Industry Applications

The applied side of everything covered in this session, at a glance:

• Real-world: odds ratios are how logistic regression results are reported and read in applied work; Minitab prints them directly while R users compute $e^{\beta }$ from the coefficients column.
• Real-world: the logit (log-odds) transformation is the standard practical trick for turning nonlinear relationships into linear ones before modeling.
• Real-world: AIC and BIC are routine model-selection criteria in time series and other applied work — compare candidates, take the lowest.
• Real-world: Fisher's scoring is the numerical algorithm behind the fitting of logistic-type models in statistical software; its iteration count is the convergence check for large data sets.
• Real-world: PCA and LDA are the core dimensionality reduction tools in machine learning and AI, where they are known as unsupervised and supervised learning techniques; PCA dates back to Hotelling in 1933, whose name also survives in the Hotelling statistic.
• Real-world: projecting high-dimensional data onto its first principal components is standard practice for visualization and for feeding downstream classifiers, with classification itself left to techniques like LDA.