Mathematical Foundations for Machine Learning

Principal Component Analysis

Published: 2026-07-11

Level: postgraduate

Audience: Postgraduate students in Mathematical Foundations for Machine Learning

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Eigenvalues and Eigenvectors — covered in Lecture 6
Eigen Decomposition (Diagonalization) — covered in Lecture 7
Constrained Optimization and Lagrange Multipliers — covered in Lecture 12

Principal Component Analysis

16.1 Introduction and Motivation

16.1.1 What PCA Solves

Hook. You have a dataset with a thousand columns — sensor readings, survey answers, image pixels. Your model runs painfully slow and barely improves with each new feature you add. What if 90% of those columns are telling you the same story, just in slightly different words? How do you find the handful of directions that actually carry the signal?

Intuition + Analogy. Think of a photographer shooting a building. She can walk around and take pictures from dozens of angles — the front, the side, a corner, the roof. But if she had to send just one photo to an architect, she'd pick the angle. That shows the most structure: the front facade with the entrance, windows. roofline visible all at once. That one photo captures the essence of the building. The other angles are redundant once you've seen this one.

PCA does exactly this for data. It finds the "best camera angle" — the direction in your high-dimensional cloud of data points that shows the most variation. Instead of looking at your data through all 100 original feature axes, PCA rotates your viewpoint. To look along the directions where the data actually spreads out.

Here's the mapping: your dataset = the building, each feature = a possible. Camera angle, the first principal component = the single best photo. The analogy breaks when the data has multiple independent directions of variation — PCA handles this. By giving you a ranked list of best angles, not just one. But the core idea holds: you trade many redundant views for a few maximally informative ones.

Formalize. PCA — Principal Component Analysis — is a linear dimensionality reduction technique. You start with a dataset of $N$ data points, each having $D$ features, stored in a matrix $X \in R^{N \times D}$ . PCA produces a new representation $Z \in R^{N \times P}$ where $P < D$ (often $P ≪ D$ ). The $P$ new dimensions — called principal components — satisfy two key properties:

They are orthogonal (uncorrelated). No principal component repeats information carried by another.
They are ordered by importance. The first principal component captures the most variance; the second captures the next most, and so on.

The key mathematical idea: among all possible linear combinations of your original features, PCA picks the ones. That maximize the spread (variance) of the projected data. Variance measures how much the data changes — and change is what carries information. A feature that is constant across all data points has zero variance and carries zero information.

When you have many features, two types of redundancy creep in: (a) near-constant features that barely vary. (b) highly correlated features that vary together. PCA strips both away automatically. It compresses your data by discarding the directions where the data barely moves, keeping only the directions of genuine variation.

Worked Example: The Kaggle Competition. Around 2014–2016, a data science team entered a Kaggle regression competition. Their baseline model used the original features and scored about 68% accuracy — not competitive. They then created PCA-transformed features from the same data. Using only the PCA features (discarding the originals), accuracy jumped to 88%. Adding the PCA features alongside the original features pushed accuracy to roughly 91%.

Let's step through what happened:

Original features: ~100 dimensions, many correlated.
PCA step: Computed the top $P$ principal components that captured ~95% of variance. This compressed the data into far fewer dimensions while preserving most of the signal.
PCA-only model: Trained on just $P$ PCA features. Accuracy: 88%.
Hybrid model: Trained on original features + PCA features. Accuracy: 91%.

The takeaway: the PCA dimensions did most of the heavy lifting — the jump. From 68% to 88% came purely from the compressed representation. The remaining 3% gain from the hybrid approach came from nuance the PCA lost.

Sense-check: If the original features were truly independent and informative, PCA would not have improved the model. The big accuracy jump confirms that most of the original features were redundant or noisy.

Assumptions & Scope. PCA relies on several assumptions. When they hold, PCA works beautifully. When they fail, results can be misleading.

Linearity. PCA finds linear combinations of features. If your data lies on a curved manifold (e.g., a spiral), PCA will flatten it poorly. Nonlinear extensions like kernel PCA or autoencoders handle curved structure.
Variance as information. PCA equates variance with information. In most settings this is reasonable, but not always. In fraud detection, the rare fraudulent transactions have low variance but are the most informative points. PCA would discard them.
Scale matters. PCA is sensitive to the units of measurement. A feature measured in millimeters has variance 1000× larger than the same feature measured in meters. Standardizing (dividing by standard deviation) before PCA fixes this.
Gaussian-ish data. PCA is not a probabilistic model — it works on any numeric data. But its variance-maximizing logic makes the most sense when the data is roughly elliptically distributed (like a multivariate Gaussian). For heavily skewed or multimodal data, interpret results cautiously.

Visual Intuition. Picture a scatter plot with axes labelled "TV ad spend" on the x-axis and "newspaper ad spend" on the y-axis. The points form a stretched, cigar-shaped cloud that runs diagonally from bottom-left. To top-right — this is what high positive correlation looks like. Now draw two arrows. The first arrow points along the long axis of the cigar. The second arrow is perpendicular to it, pointing across the narrow waist. The length of the first arrow is the variance captured by PC1. The length of the second arrow is the variance captured by PC2. Most of the points' spread — maybe 95% — happens along that first arrow. Projecting everything onto that single arrow reduces your 2D problem to 1D while keeping almost all the information. The takeaway: when features are correlated, the data lives on a lower-dimensional "shadow" of the full feature space.

Pitfalls.

Thinking PCA "understands" your features. PCA is blind to what the columns mean. It cannot tell "age" from "zip code." It just sees numbers. If you have a column of random noise with huge variance, PCA will treat it as the most important feature.
Using PCA on unstandardized data. If one feature has a range of 0–100,000. And another has a range of 0–1, PCA will be dominated entirely by the first feature. Always mean-center and standardize (or at least mean-center) before PCA.
Treating PCA as a magic bullet. PCA reduces dimensions only when there is redundancy to eliminate. If all your features are genuinely independent and equally informative, the eigenvalues will all be roughly equal. PCA cannot help in that case.
Confusing principal components with original features. PC1 is a weighted sum of all original features, not a single selected feature. You lose the physical interpretation ("salary increases by X per year of experience") in exchange for predictive power.

Student Q&A — Deduplicated. Several students asked about the practical role of PCA in a machine learning workflow:

Q: Is PCA something you do as part of feature engineering before building any model?

A: Yes, PCA is a powerful feature engineering technique. It creates new dimensions as linear combinations of your existing features. You can build your model on the PCA-reduced features alone, or combine original features with PCA features. The Kaggle example above shows the hybrid approach can outperform either pure strategy.

Q: Do we always use PCA for just a couple of features, or only when we have hundreds or thousands?

A: PCA works best when you have many dimensions. The simple 2D examples in this lecture are to help you understand the geometry. In practice, you may have 100 dimensions and find that 10 principal components capture 90% of the variance. You then train your model on those 10 instead of all 100.

Q: If we reduce from 2D to 1D, why do we call the result one-dimensional. When we still need two coordinates to plot the points?

A: Think of the transformation $Y = a X_{1} + b X_{2}$ . Two original features combine into a single value $Y$ . That $Y$ is one-dimensional — it is a single number per data point. After projection, each point lives on the new axis, not on the original $(X_{1}, X_{2})$ plane. The coordinates of that axis happen to be defined in the original 2D space. the data itself now occupies a 1D line.

Recap + Bridge. PCA finds the directions of maximum variance in your data. It compresses many redundant features into a few uncorrelated ones. The first principal component is the single direction. That captures the most spread — think of it as the best camera angle for your dataset. In the next section, we look at the two specific types of redundancy PCA exploits: constant features and correlated features.

Real-World & Domain Connection. Biologists at conservation organizations use PCA on aerial imagery to classify whale health. From photographs — the same logic the professor used in the whale camera analogy. Beyond wildlife monitoring, PCA compresses high-dimensional sensor data in satellites (hyperspectral imaging), reduces the dimensionality of gene expression datasets in bioinformatics. (where you may have 20,000 genes but only 100 patients). serves as a preprocessing step in quantitative finance where hundreds of economic indicators are distilled. Into a few "factor" components that drive market movements. In every case, the principle is the same: find the few directions that matter in a sea of correlated measurements.

16.2 Two Types of Redundancy

16.2.1 Constant Features

Hook. Imagine a weather station with 50 sensors. One sensor has been stuck at 22°C for the last three years. Can you use that sensor to predict tomorrow's temperature? No — it has never changed. It tells you absolutely nothing about how temperatures vary. Yet it sits in your dataset, taking up space, slowing down your model, and contributing zero information.

Intuition + Analogy. Think of a group of detectives trying to solve a case. Each detective brings one piece of information. One detective just repeats "it happened in Delhi" for every single case — same city, always. That detective is useless. You do not need his report; it contains no variation between cases. PCA works the same way: a column of your data matrix. Where every entry is identical (or nearly identical) is a "stuck detective." PCA ignores it.

Formally: if a column of $X$ has variance zero, it never changes. Since PCA maximizes variance, directions with zero variance get zero weight. They are discarded automatically.

Formalize. The first type of redundancy PCA eliminates is constant (or near-constant) features. A feature $X_{j}$ with sample variance

$Var (X_{j}) = \frac{1}{N - 1} i = 1 \sum N (x_{ij} - x ˉ_{j})^{2}$

that is zero (or extremely small) contains no useful signal. When you have thousands of features, the chance. That many have near-zero variance is high — especially in domains like genomics (most genes are not expressed in a given tissue). Or text analysis (most words never appear in a given document). PCA's covariance matrix $C$ captures this: near-zero diagonal entries correspond to near-constant features. The eigenvalue decomposition naturally gives these dimensions negligible weight.

16.2.2 Correlated Features

Formalize. The second type of redundancy is correlation — when two or more features move together. If $X_{1}$ and $X_{2}$ have a correlation coefficient close to $\pm 1$ , then knowing $X_{1}$ gives you almost all the information in $X_{2}$ . The model gains nothing from having both.

Concrete examples:

Age and years of driving experience: highly correlated (unsurprisingly).
Number of cylinders and engine weight in a car: more cylinders means a heavier engine block.
TV ad spend and newspaper ad spend for a company: both go up. When the marketing budget increases, both go down when it shrinks.

In PCA, correlation appears in the off-diagonal entries of the covariance matrix $C$ . A large off-diagonal entry $Cov (X_{i}, X_{j})$ signals that features $i$ and $j$ are redundant. PCA compresses this redundancy into a single new axis aligned with the direction of joint variation.

16.2.3 Visualizing Correlation with Scatter Plots

Visual Intuition. Plot TV advertising spend on the x-axis and newspaper advertising spend on the y-axis. Each data point is one company. When marketing budgets are big, both numbers go up. When budgets are small, both go down. The points stretch along a diagonal line.

Imagine the cloud of points as a cigar shape. The long axis of the cigar runs at roughly 45 degrees. This is the direction of maximum variance — where the data spreads out the most. The short axis, perpendicular to it, has minimal spread. If correlation were exactly 100%, the data would collapse perfectly onto a straight line. the variance in the perpendicular direction would be exactly zero.

This is the geometric insight that powers PCA:. When features are correlated, the data lives mostly on a lower-dimensional "shadow" of the full feature space. PCA's job is to find that shadow's axes.

The takeaway: correlation = the data cloud is stretched thin along some diagonals and narrow along others. PCA finds those diagonals and keeps them; it discards the narrow ones.

Worked Example: Salary Prediction. You are building a model to predict salary. Your dataset has these columns: age, years of experience, city, and education level. Every row has city = "Delhi." The city column is constant — it has variance zero. PCA drops it immediately. Next, age and years of experience have a correlation of about 0.95. PCA detects this from the off-diagonal entry in the covariance matrix. And merges their information into a single component — essentially a "career seniority" axis. From 4 original features, you might end up with 2 meaningful principal components. Your model now trains faster on half the columns without losing predictive power.

Sense-check: If the city column varied (Mumbai, Delhi, Bangalore), PCA would keep it. The reduction happens only where genuine redundancy exists.

Assumptions & Scope.

Correlation is linear. PCA captures linear correlation via the covariance matrix. Two features could be perfectly related by a nonlinear pattern (e.g., $X_{2} = X_{1}^{2}$ ) and still have zero linear correlation. PCA would not detect this redundancy. For nonlinear dependencies, consider kernel PCA or mutual-information-based feature selection.
Variance scale sensitivity. Imagine a feature "weight in grams" (variance ~10,000) vs. the same feature "weight in kilograms" (variance ~0.01). PCA would dramatically favor the grams version. This is why standardization before PCA is critical — it puts every feature on equal footing.
Zero variance vs. near-zero variance. In theory, PCA removes only true zero-variance features. In practice, features with variance near zero are also effectively useless. The eigenvalue spectrum shows which components contribute negligibly.

Pitfalls.

Removing features manually instead of using PCA. You might think: "I'll just compute all pairwise correlations. And drop one from each highly correlated pair." For 100 features, that is $(2 100) = 4950$ correlation checks. Then you must decide which feature to keep from each pair — an arbitrary and tedious process. PCA handles this automatically in one pass through the covariance matrix.
Confusing correlation with causation. High correlation means features move together; it does not mean one causes the other. PCA does not care about causation — it treats all features symmetrically.
Assuming constant features are rare. In high-dimensional datasets (genomics, text, images), a huge fraction of features often have near-zero variance. These are not bugs in your data — they are expected. PCA handles them gracefully.

Student Q&A:

Q: How does PCA determine which dimensions maximize the variance? Is there a mathematical formulation?

A: Yes. This is exactly what we derive in sections 16.3 through 16.5. PCA casts the problem as a constrained optimization: find the unit vector $w$ that maximizes $w^{⊤} Cw$ subject to $w^{⊤} w = 1$ . The solution uses Lagrange multipliers, and it turns out that $w$ must be an eigenvector of the covariance matrix $C$ . The eigenvalue equals the variance captured. The full derivation follows.

Recap + Bridge. PCA attacks two kinds of dead weight in your data: features. That never change (zero variance) and features that move in lockstep (high correlation). Both carry no new information. In the scatter plot of correlated features, the data stretches along a diagonal — that diagonal is the direction of maximum variance. Next, we formalize how to find that direction mathematically: projection onto a unit vector and maximizing the variance of the projected data.

Real-World & Domain Connection. In genomics, a typical microarray dataset has ~20,000 gene expression measurements but only ~100 patient samples. The vast majority of those 20,000 genes show negligible variation across patients — they are "housekeeping genes" expressed. At constant levels regardless of disease state. PCA compresses this down to a few dozen components. That capture the biologically meaningful variation, enabling clustering of cancer subtypes or drug-response groups. The same principle applies in finance: out of hundreds of economic indicators tracked. By central banks, most are strongly correlated (they all dip during recessions, rise during booms). PCA distills them into a handful of "factor" components — the underlying economic forces that actually drive the variation.

16.3 Projection and Variance Maximization

16.3.1 The Unit Vector Direction

Hook. You are standing in a dark room with a flashlight. On the wall in front of you is a cloud of glowing dots — your data. The flashlight only casts a line of light, not a cone. You can rotate it to any angle. At which angle do the dots cast the longest shadow? That angle is your first principal component. The length of the shadow is the variance it captures.

Intuition + Analogy. Think of compressing a spring. When you compress it along its natural axis, it squishes a lot — maximum change, maximum information. When you compress it from the side, it barely budges. PCA is the search for the spring's natural axis: the direction where pushing on the data produces the most movement.

A direction in $D$ -dimensional space is defined by a unit vector — a vector with length exactly 1. In 2D, the set of all unit vectors forms the unit circle (360 degrees of possible angles). In 3D, they form the unit sphere. In $D$ dimensions, they form the unit hypersphere. PCA searches this hypersphere for the one direction where projecting the data yields the greatest spread. Every candidate direction is just an arrow of length 1 pointing somewhere in your feature space.

The analogy breaks in one way: unlike a spring, data can have multiple independent directions of high variance. PCA handles this by finding all of them, ranked.

16.3.2 The Projection Formula

Formalize. You have $N$ data points, each a vector $x_{i} \in R^{D}$ (a row of your data matrix). To project data point $x_{i}$ onto a direction given by unit vector $w \in R^{D}$ (where $w^{⊤} w = 1$ ), you compute the dot product:

$z_{i} = x_{i}^{⊤} w$

The dot product geometrically gives the length of the shadow that $x_{i}$ casts onto the axis defined by $w$ . If $x_{i}$ points in the exact same direction as $w$ , $z_{i}$ equals the full length of $x_{i}$ . If $x_{i}$ is perpendicular to $w$ , $z_{i} = 0$ — it casts no shadow at all.

The result $z_{i}$ is a scalar — a single number. It is the coordinate of the projected point on the new axis. For all $N$ data points at once, stack them into the data matrix $X \in R^{N \times D}$ :

$z = Xw$

where $z \in R^{N}$ is the vector of all projected values. Each row $i$ of $z$ is $z_{i}$ . Matrix multiplication does all $N$ dot products in one operation.

Worked Example: Projection in 2D. Take a single data point $x_{1} = [3, 4]^{⊤}$ and a unit vector $w = [0.6, 0.8]^{⊤}$ . Verify $w$ is a unit vector: $0. 6^{2} + 0. 8^{2} = 0.36 + 0.64 = 1$ .

The projection is: $z_{1} = [3, 4] \cdot [0.6, 0.8] = 3 \times 0.6 + 4 \times 0.8 = 1.8 + 3.2 = 5$

The projected point lives at coordinate 5 on the axis defined by $w$ . Now take a second point $x_{2} = [1, - 1]^{⊤}$ . Its projection: $z_{2} = 1 \times 0.6 + (- 1) \times 0.8 = 0.6 - 0.8 = - 0.2$

After projection, $z = [5, - 0.2]^{⊤}$ . Your 2D data is now 1D — each point has a single coordinate on the $w$ axis.

Sense-check: The projection of $[3, 4]$ onto $[0.6, 0.8]$ gives 5. Since $∥ x_{1} ∥ = 9 + 16 = 5$ and $x_{1}$ happens to point in the exact same direction. As $w$ (check: $x_{1} /∥ x_{1} ∥ = [3/5, 4/5] = [0.6, 0.8] = w$ ), the full length is preserved — as expected.

16.3.3 Maximizing Variance after Projection

Formalize. After projecting onto a direction $w$ , we measure the spread of the projected values $z$ . Spread is quantified by the sample variance:

$Var (z) = \frac{1}{N - 1} i = 1 \sum N (z_{i} - z ˉ)^{2}$

The goal of PCA: among all unit vectors $w$ , find the one that maximizes $Var (z)$ . We do not guess. We solve:

$w max Var (Xw) subject to w^{⊤} w = 1$

Geometrically, for a dataset with high correlation (like TV spend vs. newspaper spend), projecting onto the diagonal axis — the direction of the. Cigar's long axis — gives projections spread far apart (high variance). Projecting onto the perpendicular axis — the short, narrow direction — gives projections clustered tightly together (low variance). The mathematics of section 16.5 will show that the optimal $w$ is an eigenvector of the covariance matrix.

16.3.4 Symbol Registry

Symbol	Meaning	LaTeX	Type
$x_{i}$	one data point (a row of the dataset)	$x_{i}$	vector in $R^{D}$
$X$	data matrix (N rows, D columns)	$X$	matrix, $N \times D$
$w$	projection direction (unit vector)	$w$	vector in $R^{D}$
$z_{i}$	projected value for data point i	$z_{i}$	scalar
$z$	vector of all projected values	$z$	vector in $R^{N}$
$D$	number of original features	$D$	scalar (integer)
$N$	number of data points	$N$	scalar (integer)

Comparison: PCA vs. Regression. The professor noted that PCA's projection looks similar. To fitting a best-fit line in linear regression — and the two are deeply connected:

Aspect	PCA	Regression
What it maximizes	Variance of projected data	—
What it minimizes	—	Squared error (residuals)
Direction of projection	Aligned with data spread	Aligned with predictor-response relationship
What is orthogonal to the line	Projection direction (shortest distance to axis)	Residuals (vertical distance)
Does it use a target $Y$ ?	No (unsupervised)	Yes (supervised)

They are two perspectives on the same geometric operation: projection onto a line. In regression, you fix the projection axis and minimize the perpendicular error. In PCA, you optimize the axis itself to maximize the spread. The math behind both involves the covariance structure of $X$ . When the features $X$ and the target $Y$ are jointly Gaussian, the regression line. And the first principal component of the joint $(X, Y)$ data align.

When to pick which: use regression when you have a target variable to predict. use PCA when you want to understand the structure of the features themselves without a target.

16.3.5 Student Questions and Answers

Q&A — Deduplicated. Several students asked about the relationship between PCA projection and regression, and about how to use PCA components:

Q: Is this the same idea as fitting a best-fit line in regression?

A: Good observation. PCA maximizes variance. Regression minimizes error — mean squared error. They are two sides of the same coin. In regression, when you project, the error is what you minimize. In PCA, the variance is what you maximize along the projection direction. See the comparison table above for the full contrast.

Q: After doing PCA, do we only work with Z1 (the first principal component) and ignore the rest?

A: You have new transformed features $Z_{1}, Z_{2}, \dots, Z_{D}$ . $Z_{1}$ has the most variance. $Z_{2}$ has less. You keep $Z_{1}$ and drop $Z_{2}$ if $Z_{1}$ captures enough variance, say 90% or 95%. If you need 95%, you might keep the first $P$ components whose cumulative variance exceeds that threshold. You work with those $P$ components, not the original features.

Pitfalls.

Forgetting the unit-vector constraint. Any direction can be scaled — $w$ and $10 w$ point the same way but give wildly different variances. The constraint $w^{⊤} w = 1$ pins down the scale. Without it, the optimization problem is meaningless (you could make the variance arbitrarily large by scaling $w$ ).
Thinking projection loses the original meaning. The projected value $z_{i}$ is a weighted sum of all original features. You cannot say " $z_{i}$ is mostly age" — it is a blend. This is the interpretability trade-off (discussed fully in section 16.8).
Confusing dot product with matrix multiplication order. The projection of a row vector $x_{i}$ . Onto $w$ is $x_{i}^{⊤} w$ (inner product), giving a scalar. Writing it the other way, $w^{⊤} x_{i}$ , gives the same result — the dot product is symmetric. But for the full data matrix, $X \in R^{N \times D}$ times $w \in R^{D}$ gives $z \in R^{N}$ . The order matters for getting dimensions right.

Recap + Bridge. To find the direction of maximum variance, PCA searches over all unit vectors. And projects the data onto each candidate. Projection uses the dot product: $z_{i} = x_{i}^{⊤} w$ . The winning direction is the one that maximizes $Var (z)$ . Before we can optimize anything, we first need to center the data — the next section explains why.

Real-World & Domain Connection. The projection operation. At the heart of PCA is the same mathematical operation used in computer graphics for rendering 3D scenes. Onto a 2D screen, in signal processing for matched filters (projecting a received signal onto a template to detect a known pattern). in natural language processing where word embeddings are projected onto sentiment axes to measure a document's emotional tone. In every case, the dot product with a carefully chosen direction extracts the one number that matters most from a high-dimensional object.

16.4 Mean Centering

16.4.1 Why Center the Data

(Symbols used in this section are defined in the symbol registry of section 16.5.2.)

Hook. You want to measure how much a crowd of people spreads out from "the center.". But some people are standing near the door. And others near the window — the crowd itself is shifted away from the origin. Before you measure spread, you first ask everyone to shift so the average person stands exactly at coordinates $(0, 0)$ . Now every person's position directly tells you how far they are from the group's center. Mean centering is asking your data to move so its center of mass sits at the origin.

Intuition + Analogy. Think of a scale (balance) at a grocery store. Before weighing anything, you press the "tare" button. To zero the scale — this subtracts the weight of the container so you measure only the contents. Mean centering is the tare button for data: you subtract the average so you measure only the deviations from average. A data point of "age = 30" becomes "age = 0 (relative to mean)" after centering — it is average. A point with "age = 40" becomes "+10" — ten years above average.

The analogy maps perfectly: the container weight = the mean vector, the net. Contents = the centered data, the measurement = variance computation. The break point: unlike a grocery scale where negative weight is impossible, centered data has both positive. And negative values — that is expected and fine.

Formalize. Mean centering transforms the data matrix $X \in R^{N \times D}$ into centered data $X_{centered}$ by subtracting the column means:

$X_{centered} = X - X ˉ$

where $X ˉ = [x ˉ_{1}, x ˉ_{2}, \dots, x ˉ_{D}]$ and each $x ˉ_{j} = \frac{1}{N} \sum_{i = 1}^{N} x_{ij}$ is the mean of column $j$ . After this operation, every column of $X_{centered}$ has mean zero:

$\frac{1}{N} i = 1 \sum N (X_{centered})_{ij} = 0 for all j = 1, \dots, D$

Why is this necessary? Two reasons:

Simplifies the variance formula. The sample variance of projected values would normally be $\frac{1}{N - 1} \sum (z_{i} - z ˉ)^{2}$ . After centering, $z ˉ = 0$ (proved below), so the formula collapses to $\frac{1}{N - 1} \sum z_{i}^{2}$ . This is much cleaner algebraically.

Geometric alignment. PCA searches for the best projection direction through the origin. If the data cloud is shifted away from the origin, projection. Through the origin gives distorted results — you are measuring spread from $(0, 0)$ instead of from the data's actual center. Centering moves the cloud so its center of mass sits exactly at the origin, making projections through the origin meaningful.

An important theoretical note: centering is done without loss of generality. The variance of the low-dimensional code does not depend on the data's mean:

$V_{z} [z] = V_{x} [B^{⊤} (x - μ)] = V_{x} [B^{⊤} x - B^{⊤} μ] = V_{x} [B^{⊤} x]$

The constant shift $B^{⊤} μ$ does not affect variance. So we can safely assume mean zero for the rest of the derivation.

16.4.2 How Mean Centering Works

Worked Example: Centering by Hand. Take a small dataset with Age and Experience:

Age	Experience
20	2
30	12
40	10

Step 1: Compute column means.

Mean Age: $(20 + 30 + 40) /3 = 90/3 = 30$
Mean Experience: $(2 + 12 + 10) /3 = 24/3 = 8$

Step 2: Subtract the mean from each entry.

Age (centered)	Experience (centered)
$20 - 30 = - 10$	$2 - 8 = - 6$
$30 - 30 = 0$	$12 - 8 = 4$
$40 - 30 = 10$	$10 - 8 = 2$

Step 3: Verify.

Mean of centered Age: $(- 10 + 0 + 10) /3 = 0$ ✓
Mean of centered Experience: $(- 6 + 4 + 2) /3 = 0$ ✓

The data is now centered at the origin. This means any projection of the data onto a direction. Through the origin — which is what PCA does — will have $z ˉ = 0$ .

Proof that $z ˉ = 0$ after centering. Let $z = X_{centered} w$ . Then:

$z ˉ = \frac{1}{N} i = 1 \sum N z_{i} = \frac{1}{N} i = 1 \sum N j = 1 \sum D (X_{centered})_{ij} w_{j} = j = 1 \sum D w_{j} (\frac{1}{N} i = 1 \sum N (X_{centered})_{ij}) = j = 1 \sum D w_{j} \cdot 0 = 0$

Every column of $X_{centered}$ has mean zero, so every term in the sum is zero. The result: $z ˉ = 0$ .

Sense-check: If you project the centered data $[(- 10, - 6), (0, 4), (10, 2)]$ onto $w = [1, 0]^{⊤}$ (pure Age direction), you get $[- 10, 0, 10]$ . The mean is 0, as expected.

Assumptions & Scope.

Centering is almost always done. In theory, centering is not strictly required. — you can subtract the mean inside the variance formula every time. But the algebra becomes much uglier, and numerical stability suffers. All standard PCA implementations center the data first.
Standardization is optional but often recommended. After centering (mean = 0), you may also divide each column. By its standard deviation (variance = 1). This is called standardization and ensures all features contribute equally. Without it, a feature measured in large units dominates PCA. The professor's examples use only centering; in practice, standardization before PCA is the default in most libraries.
Centering does not change the geometry of the data. It is a rigid translation — the shape, correlations. relative distances between points are preserved. Only the coordinate origin moves.

Visual Intuition. Picture a 2D scatter plot of Age on the x-axis and Experience on the y-axis. Before centering, the points cluster around $(30, 8)$ . The axes cross at $(0, 0)$ , far from the data. After centering, the points stay in exactly the same arrangement relative to one another. the whole cloud shifts so its center sits at $(0, 0)$ . The axes now pass through the middle of the data. Any rotation of these axes (which is what PCA does) now makes geometric. Sense — you are rotating around the data's natural center.

Pitfalls.

Applying centering to the test set using the test set's own mean.. When you deploy a PCA model, the test data must be centered using the training data's mean, not its own mean. Otherwise, the centering is inconsistent and the PCA projection will be wrong. This is the same rule as in standardization for any ML model: fit on training, apply to test.
Forgetting to center at all. If you skip centering. And compute $X^{⊤} X$ , you get a matrix that mixes the mean with the covariance. The eigenvectors will be pulled toward the mean vector, not toward the directions of genuine variation. The results are misleading.
Assuming centering removes all offset issues. If your data has outliers, the mean itself can be pulled away. From the true center of the data. In such cases, robust alternatives like the median may be more appropriate (though not standard in PCA).

16.4.3 Student Questions and Answers

Q&A — Deduplicated. Multiple students asked why $z ˉ = 0$ after centering:

Q: Why is $z ˉ = 0$ after we mean-center $X$ ?

A: Because $z = X_{centered} w$ . The mean of $z$ is:

$z ˉ = \frac{1}{N} i = 1 \sum N z_{i} = \frac{1}{N} i = 1 \sum N (x_{i}^{centered})^{⊤} w$

Since the sum is linear, you can pull $w$ out:

$z ˉ = (\frac{1}{N} i = 1 \sum N x_{i}^{centered})^{⊤} w = 0^{⊤} w = 0$

Every column of $X_{centered}$ has mean zero. the mean vector itself is the zero vector. its dot product with anything is zero. The key insight: a linear combination of zero-mean quantities is itself zero-mean.

Recap + Bridge. Mean centering shifts the data so its center of mass sits at the origin. This simplifies the variance to a sum-of-squares ( $z ˉ = 0$ ) and aligns the geometry so projecting through the origin is meaningful. The T1 textbook confirms this is done without loss of generality — the variance of projected data is invariant. To a constant shift. Now that the data sits at the origin, we can build the full mathematical machinery: the covariance matrix. And the eigenvalue optimization that solves PCA.

Real-World & Domain Connection. Mean centering is so fundamental. That it appears under different names across fields: "baseline correction" in spectroscopy (subtracting. The background signal), "demeaning" in econometrics (removing individual fixed effects in panel data). "DC offset removal" in signal processing (centering an audio waveform around zero before Fourier analysis). The principle is always the same: remove the constant component so you can analyze the variation.

16.5 Mathematical Formulation

16.5.1 Variance of Projected Data

Formalize — Full Derivation (Part 1: From Variance to Covariance).

We start with the variance of the projected data $z = Xw$ . For any set of numbers, the sample variance measures how much they spread around their mean:

$Var (z) = \frac{1}{N - 1} i = 1 \sum N (z_{i} - z ˉ)^{2} (1)$

After mean centering (section 16.4), we proved that $z ˉ = 0$ . This collapses the formula to a pure sum of squares:

$Var (z) = \frac{1}{N - 1} i = 1 \sum N z_{i}^{2} (2)$

In vector form, the sum of squared entries of $z$ is the dot product of $z$ with itself:

$Var (z) = \frac{1}{N - 1} z^{⊤} z (3)$

Now substitute the projection formula $z = Xw$ :

$Var (z) & = \frac{1}{N - 1} (Xw)^{⊤} (Xw) & = \frac{1}{N - 1} w^{⊤} X^{⊤} Xw (transpose rule: (AB)^{⊤} = B^{⊤} A^{⊤}) (4)$

The expression $w^{⊤} X^{⊤} Xw$ is a quadratic form — it computes a scalar that depends quadratically on $w$ . This scalar is proportional to the variance captured by direction $w$ .

Dimensional check: $w \in R^{D}$ , $X \in R^{N \times D}$ , so $Xw \in R^{N}$ , $(Xw)^{⊤} \in R^{1 \times N}$ , and the product is $1 \times 1$ — a scalar. ✓

16.5.2 Symbol Registry

Symbol	Meaning	LaTeX	Type
$x_{i}$	data point (row vector)	$x_{i}$	vector in $R^{D}$
$X$	data matrix (N rows, D columns)	$X$	matrix, $N \times D$
$w$	projection direction / unit vector	$w$	vector in $R^{D}$
$z_{i}$	projected scalar value for point i	$z_{i}$	scalar
$z$	vector of all projected values	$z$	vector in $R^{N}$
$C$	covariance matrix	$C$	matrix, $D \times D$
$λ$	eigenvalue	$λ$	scalar
$z ˉ$	mean of projected values	$z ˉ$	scalar
$N$	number of data points	$N$	scalar (integer)
$D$	number of original features	$D$	scalar (integer)

16.5.3 The Covariance Matrix

Formalize — Full Derivation (Part 2: The Covariance Matrix).

The term $\frac{1}{N - 1} X^{⊤} X$ is the sample covariance matrix of the centered data, denoted $C$ :

$C = \frac{1}{N - 1} X^{⊤} X \in R^{D \times D} (5)$

This matrix encodes the entire second-order structure of your data. Every entry tells you how two features relate:

Diagonal entries $C_{j j} = \frac{1}{N - 1} \sum_{i} (x_{ij})^{2}$ give the variance of feature $j$ .
Off-diagonal entries $C_{j k} = \frac{1}{N - 1} \sum_{i} x_{ij} x_{ik}$ give the covariance between features $j$ and $k$ .

The covariance matrix is symmetric ( $C_{j k} = C_{k j}$ ) because $Cov (X_{j}, X_{k}) = Cov (X_{k}, X_{j})$ . It is also positive semidefinite — all its eigenvalues are $\geq 0$ , a fact that guarantees variances never come out negative.

Substituting $C$ into the variance expression gives the compact form:

$Var (z) = w^{⊤} Cw (6)$

This is a quadratic form. For any direction $w$ , this scalar tells you exactly how much variance you capture by projecting onto that direction.

Why is $C$ always square? $X$ is $N \times D$ (tall, many rows, few columns). But $X^{⊤} X$ is $(D \times N) (N \times D) = D \times D$ — always square. This is crucial because eigen decomposition requires a square matrix. You do eigen decomposition on $C$ , never on $X$ .

16.5.4 Constrained Optimization

Formalize — Full Derivation (Part 3: The Optimization Problem).

We now have a crisp optimization problem:

$w max w^{⊤} Cw subject to w^{⊤} w = 1 (7)$

The constraint $w^{⊤} w = 1$ is essential. Without it, you could make the variance arbitrarily large simply by scaling $w$ — the direction would be the same. the "variance" would inflate to infinity. The unit-vector constraint pins the scale: we are searching over directions, not magnitudes.

This is a constrained optimization problem of the exact form studied in the previous lecture on duality. The professor explicitly designed the course flow so that constrained optimization theory (Lagrange multipliers, KKT conditions) would be immediately applicable to PCA.

16.5.5 Lagrangian Formulation

Formalize — Full Derivation (Part 4: Lagrangian and Derivative).

To solve (7), we form the Lagrangian by introducing a multiplier $λ$ for the equality constraint:

$L (w, λ) = w^{⊤} Cw + λ (w^{⊤} w - 1) (8)$

Notation note. The professor writes the Lagrangian with $+ λ$ . Some textbooks use $- λ$ for maximization problems. The sign convention does not matter — it gets absorbed when we re-label $λ$ after differentiation (see below). The T1 textbook (Deisenroth et al.) uses the identical form: $L (b_{1}, λ) = b_{1}^{⊤} S b_{1} + λ_{1} (1 - b_{1}^{⊤} b_{1})$ , which is equivalent up to a sign flip.

Now compute the gradient with respect to $w$ and set to zero:

$\frac{\partial L}{\partial w} = 2 Cw + 2 λ w = 0 (9)$

Here's why:

$\frac{\partial}{\partial w} (w^{⊤} Cw) = 2 Cw$ because $C$ is symmetric. In the 2D case: $w = [v_{1}, v_{2}]^{⊤}$ , $C = [c_{11} & c_{12} c_{12} & c_{22}]$ , and $w^{⊤} Cw = c_{11} v_{1}^{2} + 2 c_{12} v_{1} v_{2} + c_{22} v_{2}^{2}$ . Differentiating: $\frac{\partial}{\partial v _{1}} = 2 c_{11} v_{1} + 2 c_{12} v_{2}$ , $\frac{\partial}{\partial v _{2}} = 2 c_{12} v_{1} + 2 c_{22} v_{2}$ , which stacks into $2 Cw$ .
$\frac{\partial}{\partial w} (λ w^{⊤} w) = 2 λ w$ because $w^{⊤} w = \sum_{j} v_{j}^{2}$ , and each derivative gives $2 v_{j}$ , stacking into $2 w$ .

From (9):

$2 Cw + 2 λ w = 0 ⟹ Cw = - λ w (10)$

The minus sign is an artifact of the $+$ sign convention in the Lagrangian. We absorb it by redefining $λ \leftarrow - λ$ . This renaming is purely cosmetic — $λ$ is just an unknown scalar we are solving for. After renaming:

$Cw = λ w (11)$

Dimensional check: $C \in R^{D \times D}$ , $w \in R^{D}$ , so $Cw \in R^{D}$ . RHS: $λ$ scalar, $λ w \in R^{D}$ . Dimensions match. ✓

16.5.6 The Eigenvalue Connection

Formalize — Full Derivation (Part 5: Eigenvalue Equation).

Equation (11) is the defining equation of an eigenvalue problem. It says: when the matrix $C$ multiplies the vector $w$ , the result is just $w$ scaled by $λ$ . This is exactly what defines an eigenvector (the direction $w$ ) and its eigenvalue (the scale factor $λ$ ).

The key insight: the optimal projection direction must be an eigenvector of the covariance matrix. PCA does not require us. To search over 360 degrees. Or over a hypersphere of directions — we simply compute the eigenvectors of $C$ and pick the one with the largest eigenvalue.

The professor introduced PCA immediately after constrained optimization and eigenvalue theory. Because PCA is the perfect application: the constrained optimization problem (maximize variance, subject. To a unit-vector constraint) naturally collapses to an eigenvalue equation. The mathematics of the previous lectures directly solves the practical problem of dimensionality reduction.

16.5.7 Variance Equals Eigenvalue

Formalize — Full Derivation (Part 6: The Punchline).

Now substitute the eigenvalue relationship $Cw = λ w$ back into the variance expression:

$Var (z) & = w^{⊤} Cw & = w^{⊤} (λ w) (substitute Cw = λ w) & = λ (w^{⊤} w) (pull scalar λ out) & = λ (since w^{⊤} w = 1) (12)$

The variance captured by projecting onto eigenvector $w$ is its eigenvalue $λ$ . This is the punchline of PCA.

Therefore:

To maximize variance, pick the eigenvector with the largest eigenvalue.
$λ_{1}$ = maximum possible variance after projection.
$v_{1}$ = the first principal component (the associated eigenvector).
$λ_{2}$ = variance captured by the second principal component (the eigenvector orthogonal to $v_{1}$ with the next largest eigenvalue).

Limiting-case check: If $C = I$ (identity matrix), all eigenvalues are 1. Every direction has variance 1 — the data is perfectly spherical with no preferred axis. PCA cannot reduce dimensions in this case. This makes sense: if all features are uncorrelated and have equal variance, there is no redundancy to eliminate. ✓

Boundary check: Since $C$ is positive semidefinite ( $C = \frac{1}{N - 1} X^{⊤} X$ ), all eigenvalues satisfy $λ \geq 0$ . A variance cannot be negative. ✓

Worked Example: Numerical Spot-Check. Let $w = [1, 0]^{⊤}$ (unit vector). Let $C = [3&1 1&2]$ . The variance along $w$ is:

$w^{⊤} Cw = [10] [3&1 1&2] [10] = [10] [31] = 3$

Now compute the eigenvalues of $C$ : $det (C - λ I) = (3 - λ) (2 - λ) - 1 = λ^{2} - 5 λ + 5 = 0$ . Solutions: $λ_{1, 2} = \frac{5 \pm 5}{2} \approx 3.618$ and $1.382$ . The maximum variance achievable by any direction is $λ_{1} \approx 3.618$ , which is greater than the 3 we got from the naive direction $[1, 0]^{⊤}$ . The eigenvector associated with $λ_{1}$ gives the truly optimal direction.

Sense-check: The maximum possible variance (3.618) is larger than the variance along either original axis (3 or 2). PCA finds a blend of the axes that captures more spread than either axis alone. ✓

Visual Intuition. Picture the covariance matrix $C$ as an ellipse. The eigenvectors are the axes of the ellipse. The eigenvalues are the lengths of those axes (specifically, the square root of the eigenvalue is the length). In 2D, $C$ defines an ellipse centered at the origin. PC1 points along the ellipse's major (longest) axis. PC2 points along the minor (shortest) axis. The ratio $λ_{1} / λ_{2}$ measures how elongated the ellipse is. A ratio near 1 means the data is nearly circular — PCA helps little. A ratio of 100 means the data is a thin cigar — PCA helps enormously.

Assumptions & Scope.

$C$ is computed from centered data. All derivations above assume $X$ has zero-mean columns. If you skip centering, $X^{⊤} X$ mixes the mean with the covariance, and the eigenvalue equation no longer gives directions of genuine variance.
$C$ is positive semidefinite. As a Gram matrix scaled by $1/ (N - 1)$ , $C$ is always PSD. This guarantees real, non-negative eigenvalues. The variance along any direction is always $\geq 0$ .
The derivation uses $\frac{1}{N - 1}$ (sample covariance). Some texts use $\frac{1}{N}$ (population covariance). The eigenvectors are identical — scaling by a constant does not change eigenvector directions. Only the eigenvalues scale by the factor $(N - 1) / N$ .

16.5.8 Student Questions and Answers

Q&A — Deduplicated. Multiple students asked about the derivative computation, the square-matrix requirement, and the Lagrangian sign:

Q: How does $\frac{\partial}{\partial w} (w^{⊤} Cw) = 2 Cw$ work?

A: Think of the 2D case. Let $w = [v_{1}, v_{2}]^{⊤}$ and $C = [c_{11} & c_{12} c_{12} & c_{22}]$ (symmetric). Expand the quadratic form:

$w^{⊤} Cw = c_{11} v_{1}^{2} + 2 c_{12} v_{1} v_{2} + c_{22} v_{2}^{2}$

Taking partial derivatives:

$\frac{\partial}{\partial v _{1}} = 2 c_{11} v_{1} + 2 c_{12} v_{2}$
$\frac{\partial}{\partial v _{2}} = 2 c_{12} v_{1} + 2 c_{22} v_{2}$

Stacking these into a vector: $[2 c_{11} v_{1} + 2 c_{12} v_{2} 2 c_{12} v_{1} + 2 c_{22} v_{2}] = 2 [c_{11} & c_{12} c_{12} & c_{22}] [v_{1} v_{2}] = 2 Cw$ .

The same pattern generalizes to any dimension. If $C$ were not symmetric, the derivative would be $Cw + C^{⊤} w$ . But the covariance matrix is always symmetric, so these two terms are equal, giving $2 Cw$ .

Q: Eigenvectors are only for square matrices. $X$ is $N \times D$ (not square when $N \neq = D$ ). How does this work?

A: You do eigen decomposition on $C = \frac{1}{N - 1} X^{⊤} X$ , not on $X$ . $C$ is $D \times D$ — always square, regardless of how many data points you have. If you have 2 features, $C$ is $2 \times 2$ . If you have 100 features, $C$ is $100 \times 100$ . It is square because it pairs every feature with every other feature.

Q: Can you explain the Lagrangian constraint sign? Shouldn't it be $\leq 0$ ?

A: The constraint $w^{⊤} w - 1 = 0$ is an equality constraint. The professor writes the Lagrangian as $L = f + λ g$ (with $g = w^{⊤} w - 1$ ). This gives $Cw = - λ w$ , and after absorbing the sign (renaming $- λ \to λ$ ), we get $Cw = λ w$ . The standard T1 textbook uses the same convention. Whether you write $+ λ$ or $- λ$ in the Lagrangian is a convention — the final. Eigenvalue equation is identical after re-labeling the multiplier. The KKT conditions (covered in the next lecture) handle the inequality-versus-equality case formally.

Pitfalls.

Confusing multiplication order. $X^{⊤} X$ is $D \times D$ (covariance). $X X^{⊤}$ is $N \times N$ (Gram matrix). They have the same nonzero eigenvalues but different eigenvectors and different sizes. PCA needs $X^{⊤} X$ (the covariance matrix), which is always $D \times D$ .
Forgetting the constraint. The derivative $\frac{\partial}{\partial w} w^{⊤} Cw = 2 Cw$ alone would be maximized by making $∥ w ∥ \to \infty$ . The constraint $w^{⊤} w = 1$ is not optional — without it, the optimization problem is unbounded.
Thinking the minus sign in $Cw = - λ w$ means $λ$ is negative. The $λ$ in the Lagrangian is just a temporary symbol. After absorbing the sign, the eigenvalues of $C$ are non-negative because $C$ is positive semidefinite. The sign flip is just bookkeeping.

Recap + Bridge. We derived the full mathematical engine of PCA: (1) express variance of projected data. As a quadratic form $w^{⊤} Cw$ , (2) maximize it subject to $w^{⊤} w = 1$ using Lagrange multipliers, (3) discover. That the solution is the eigenvector with the largest eigenvalue. (4) prove that the eigenvalue is the variance captured. The $1/ (N - 1)$ factor is sample covariance; some texts use $1/ N$ (population). The eigenvectors are identical either way. Next, we turn this math into a concrete five-step algorithm and compute a full numerical example by hand.

Real-World & Domain Connection. The eigenvalue-eigenvector framework behind PCA is the same mathematical machinery used. By Google's original PageRank algorithm (the principal eigenvector of the web's link matrix ranks pages. By importance), by structural engineers analyzing vibration modes of bridges and buildings (eigenvalues give natural frequencies. eigenvectors give mode shapes). by quantum mechanics (the Schr\u00f6dinger equation is an eigenvalue problem for the Hamiltonian operator). PCA applies exactly the same linear algebra to data instead of web links or buildings.

16.6 The PCA Algorithm

16.6.1 Step-by-Step Procedure

(All symbols used in this section are fully defined in the symbol registry of section 16.5.2.)

Exam note: PCA was an assignment topic last semester, so it was excluded from the exam. This year PCA is not an assignment — it is very likely. To appear in the examination. Expect a numerical problem requiring you to center data, compute the covariance matrix, find eigenvalues. identify the direction of maximum variance.

Purpose. The PCA algorithm takes a high-dimensional dataset and produces a low-dimensional representation that preserves as much variance (information) as possible. It solves the dual problems of redundancy (correlated features) and noise (near-constant features) by rotating the data. Into a new coordinate system where axes are ordered by importance and are mutually uncorrelated.

Inputs & Outputs.

	Description	Shape
Input: $X$	Data matrix (can be uncentered)	$N \times D$
Hyperparameter: $P$	Number of components to keep (or variance threshold, e.g., 95%)	scalar
Output: $Z$	Reduced representation (projected data)	$N \times P$
Output: $V$	Matrix of top $P$ eigenvectors (columns)	$D \times P$
Output: $λ$	Corresponding eigenvalues	$P$ scalars

Steps. The PCA algorithm in five steps:

Why: Variance is measured around the mean. Centering simplifies the covariance computation and makes projection through the origin meaningful.

Center the data. Compute the mean of each column: $x ˉ_{j} = \frac{1}{N} \sum_{i = 1}^{N} x_{ij}$ . Subtract: $X_{centered} = X - X ˉ$ . Every column now has mean zero.

Why: $C$ captures all pairwise relationships between features. Its eigenvectors will be the principal directions.

Compute the covariance matrix. $C = \frac{1}{N - 1} X_{centered}^{⊤} X_{centered}$ . This matrix is $D \times D$ , square and symmetric.

Why: We proved in section 16.5 that the eigenvectors are the optimal projection directions. the eigenvalues equal the variance captured by each.

Eigen-decompose $C$ . Solve $C v_{k} = λ_{k} v_{k}$ for $k = 1, \dots, D$ . You get $D$ eigenvectors $v_{1}, \dots, v_{D}$ (each a $D$ -dimensional unit vector) and $D$ eigenvalues $λ_{1}, \dots, λ_{D}$ (non-negative scalars).

Why: The ranking tells you which directions matter most.

Sort by descending eigenvalue. Reorder so $λ_{1} \geq λ_{2} \geq \dots \geq λ_{D} \geq 0$ . The first eigenvector $v_{1}$ is PC1 (direction of maximum variance), $v_{2}$ is PC2 (orthogonal to $v_{1}$ , next most variance), and so on.

Choose $P$ and project. Select how many components to keep — either a fixed $P$ . Or enough to reach a cumulative variance threshold (e.g., 95%). Form the projection matrix $V_{P} = [v_{1}, \dots, v_{P}] \in R^{D \times P}$ . Compute the reduced data:

$Z = X_{centered} V_{P} \in R^{N \times P}$

Why: $Z$ is your new, compressed dataset. Each row is an $N$ -point in $P$ -dimensional space. The columns $Z_{1}, \dots, Z_{P}$ are your new features — uncorrelated and ordered by importance.

16.6.2 Proportion of Variance Explained

The variance along an eigenvector equals its eigenvalue. The total variance in the data is the sum of all eigenvalues — every direction's variance combined:

$Total variance = k = 1 \sum D λ_{k} = tr (C)$

The proportion captured by the first $P$ components is:

$Variance retained = \frac{\sum _{k = 1}^{P} λ _{k}}{\sum _{k = 1}^{D} λ _{k}}$

For the 2D case, the proportion along PC1 alone is:

$\frac{λ _{1}}{λ _{1} + λ _{2}}$

Example: If $λ_{1} = 40$ and $λ_{2} = 1$ , then $40/41 \approx 97.6%$ . You keep only PC1 because it captures nearly 98% of all the information. The 2.4% you lose is the variance in the narrow, perpendicular direction — which in this case is negligible.

In practice, you plot the cumulative proportion against the number of components (a scree plot). And choose $P$ where the curve levels off. A common threshold is 95% — keep enough components to explain 95% of total variance.

Trace: Tiny Dataset Walkthrough. Consider this dataset of 3 points with 2 features (we work the full numerical example in section 16.7. here we trace the procedure):

$X = 1&1 2&3 3&5, N = 3, D = 2$

Step 1 — Center: Column means are $x ˉ_{1} = 2$ , $x ˉ_{2} = 3$ . Centered: $X_{centered} = - 1& - 2 0&0 1&2$

Step 2 — Covariance: $C = \frac{1}{2} [- 1&0&1 - 2&0&2] - 1& - 2 0&0 1&2 = [1&2 2&4]$

Step 3 — Eigen-decompose: $det (C - λ I) = λ (λ - 5) = 0 ⟹ λ_{1} = 5, λ_{2} = 0$ .

Step 4 — Sort: Already $λ_{1} > λ_{2}$ .

Step 5 — Choose $P$ : Proportion = $5/5 = 100%$ . Keep $P = 1$ . Project onto $v_{1}$ .

Result: The data, originally 2D, collapses perfectly to 1D. With zero information loss — because the second column was exactly $2 \times$ the first. PCA detected and eliminated that perfect redundancy.

Complexity & Cost. Computing the covariance matrix costs $O (N D^{2})$ — for each of the $D (D + 1) /2$ unique entries, you sum over $N$ points. Full eigen decomposition costs $O (D^{3})$ using standard algorithms. Total: $O (N D^{2} + D^{3})$ .

In practice, when $N ≪ D$ (fewer data points than features, common in genomics), you can swap. To the $N \times N$ Gram matrix trick: compute eigenvectors of $\frac{1}{N - 1} X X^{⊤}$ (size $N \times N$ , cost $O (N^{3})$ ) and then recover the $D$ -dimensional eigenvectors. This is what Python's sklearn.decomposition.PCA does internally via SVD, which handles both cases efficiently. For extremely large datasets, randomized PCA or incremental PCA (processing data in batches) is used.

When to Use / Alternatives.

Use PCA when: you have many features, suspect high correlation, and want to speed up training or visualize high-dimensional data. PCA is also excellent as a preprocessing step before clustering (k-means on PCA-reduced data often works better. Because Euclidean distance is more meaningful in lower dimensions).
Do not use PCA when: you need interpretable features (PCA components are opaque linear blends),. When all features are independent and equally important (eigenvalues will be similar — PCA cannot help). when the data has strong nonlinear structure (try kernel PCA, t-SNE. UMAP instead).
Alternatives: SVD (mathematically equivalent, numerically preferred for large data), Factor Analysis (probabilistic model. With per-feature noise variances), Independent Component Analysis (ICA — finds statistically independent, not just uncorrelated, components), t-SNE/UMAP (nonlinear, better. For visualization but not invertible).

16.6.3 Student Questions and Answers

Q: What happens if the eigenvalues are all similar? How do you decide how many to keep?

A: When eigenvalues are similar — say 2.1, 1.8, and 1.9 — every direction is roughly equally important. You cannot reduce the dimensionality meaningfully. The scree plot will be a flat line with no clear elbow. You end up needing all the original dimensions. PCA cannot help when features are distinct and uncorrelated.

Q: Do we have to compute the covariance matrix manually for numerical exam problems?

A: In the exam, you may be given a small matrix and asked to find the direction of maximum variance. You will need to: (1) center the data, (2) compute the covariance matrix, (3) find eigenvalues. By solving the characteristic equation $det (C - λ I) = 0$ , (4) identify the largest eigenvalue, (5) optionally find the corresponding eigenvector. The next section (16.7) walks through a complete numerical example you should study carefully.

Pitfalls.

Using $\frac{1}{N}$ instead of $\frac{1}{N - 1}$ . The sample covariance uses $N - 1$ (Bessel's correction) to give an unbiased estimate. Using $N$ scales all eigenvalues by $(N - 1) / N$ but does not change eigenvectors or the proportion of variance. For exam purposes, use whichever the professor specifies.
Forgetting to center the data. In a hurry, you might compute $X^{⊤} X$ without centering first. This is wrong — it mixes the mean with the covariance. The professor specifically tests this step.
Picking $P$ arbitrarily. Choosing $P = 2$ because "2D plots are nice" is not a statistical criterion. Use the cumulative variance threshold or the scree plot elbow. If you must pick a number, justify it with the proportion of variance retained.
Applying PCA blindly to categorical variables. PCA works on continuous numeric data. Binary or categorical features encoded as 0/1 can technically be PCA'd. the covariance of binary variables often violates the assumptions and produces misleading components.

Recap + Bridge. PCA is a five-step algorithm: center, compute covariance, eigen-decompose, sort by eigenvalue, project onto the top $P$ . The proportion of variance retained is $\sum_{k = 1}^{P} λ_{k} / \sum_{k = 1}^{D} λ_{k}$ . When features are redundant, a small $P$ captures nearly everything. When eigenvalues are all similar, PCA cannot help. Next, we work through a full numerical example by hand — computing every matrix multiplication, determinant, and eigenvalue exactly.

Real-World & Domain Connection. The standard PCA algorithm described here is implemented. In virtually every data science library: sklearn.decomposition.PCA in Python, prcomp in R, pca in MATLAB. These implementations use SVD under the hood for numerical stability. PCA also serves as the backbone of eigenfaces in computer vision — a facial recognition technique. That applies PCA to a database of face images, representing each face as a weighted sum of "eigenface" components. The top eigenfaces capture lighting variation, head shape, and major facial features; discarding low-variance components filters out expression noise and pixel-level artifacts.

16.7 Worked Numerical Example

(All symbols used in this numerical walkthrough are defined in the symbol registry of section 16.5.2.)

16.7.1 Setting Up the Data

Full PCA Calculation by Hand. We work through every step of PCA on a tiny dataset. Follow along with pencil and paper — the exam will test exactly this procedure with different numbers.

$X = 1&1 2&3 3&5, N = 3, D = 2$

Three houses. Two features: $X_{1}$ could be square footage, $X_{2}$ could be number of rooms. Notice that $X_{2}$ seems to grow with $X_{1}$ . Let's see what PCA reveals.

16.7.2 Mean Centering

Step 1: Center the data.

Column means:

$x ˉ_{1} = (1 + 2 + 3) /3 = 2$
$x ˉ_{2} = (1 + 3 + 5) /3 = 3$

Subtract the means from every entry:

$X_{1}$ (centered)	$X_{2}$ (centered)
$1 - 2 = - 1$	$1 - 3 = - 2$
$2 - 2 = 0$	$3 - 3 = 0$
$3 - 2 = 1$	$5 - 3 = 2$

$X_{centered} = - 1& - 2 0&0 1&2$

Observation: The second column is exactly 2 times the first column: $(- 1, 0, 1) \times 2 = (- 2, 0, 2)$ . The two features are 100% linearly dependent. This means one feature is completely redundant — PCA should detect this and tell us we only need 1 dimension.

16.7.3 Computing the Covariance Matrix

Step 2: Compute $C$ . The formula uses $N - 1 = 2$ :

$C = \frac{1}{N - 1} X_{centered}^{⊤} X_{centered}$

First compute $X^{⊤} X$ :

$X^{⊤} X = [- 1&0&1 - 2&0&2] - 1& - 2 0&0 1&2$

Multiply element by element:

Top-left (1,1): $(- 1) (- 1) + (0) (0) + (1) (1) = 1 + 0 + 1 = 2$
Top-right (1,2): $(- 1) (- 2) + (0) (0) + (1) (2) = 2 + 0 + 2 = 4$
Bottom-left (2,1): $(- 2) (- 1) + (0) (0) + (2) (1) = 2 + 0 + 2 = 4$
Bottom-right (2,2): $(- 2) (- 2) + (0) (0) + (2) (2) = 4 + 0 + 4 = 8$

$X^{⊤} X = [2&4 4&8]$

Now divide by $N - 1 = 2$ :

$C = \frac{1}{2} [2&4 4&8] = [1&2 2&4]$

Check: The diagonal $C_{11} = 1$ is the variance of $X_{1}$ after centering: $((- 1)^{2} + 0^{2} + 1^{2}) /2 = 2/2 = 1$ . The diagonal $C_{22} = 4$ is the variance of $X_{2}$ : $((- 2)^{2} + 0^{2} + 2^{2}) /2 = 8/2 = 4$ . The off-diagonal $C_{12} = 2$ is the covariance: $((- 1) (- 2) + 0 + (1) (2)) /2 = 4/2 = 2$ . All consistent.

16.7.4 Eigen Decomposition

Step 3: Find eigenvalues. Solve the characteristic equation $det (C - λ I) = 0$ :

$det ([1 - λ &2 2&4 - λ]) = (1 - λ) (4 - λ) - (2) (2) = 0$

Expand: $(1 - λ) (4 - λ) - 4 = (4 - λ - 4 λ + λ^{2}) - 4 = λ^{2} - 5 λ + 4 - 4 = λ^{2} - 5 λ = 0$

The 4 and -4 cancel nicely. Factor: $λ (λ - 5) = 0 ⟹ λ_{1} = 5, λ_{2} = 0$

Step 4: Find eigenvectors. For each eigenvalue, solve $(C - λ I) v = 0$ .

For $λ_{1} = 5$ : $(C - 5 I) = [1 - 5&2 2&4 - 5] = [- 4&2 2& - 1]$

Both rows give the same equation (the second row is $- 1/2$ times the first): $- 4 v_{1} + 2 v_{2} = 0 ⟹ v_{2} = 2 v_{1}$ . Choose $v_{1} = 1$ , so $v_{2} = 2$ . The raw eigenvector is $[1, 2]^{⊤}$ .

Normalize to a unit vector: $∥ v_{1} ∥ = 1^{2} + 2^{2} = 5$ .

$v_{1} = \frac{1}{5} [12] \approx [0.447 0.894]$

For $λ_{2} = 0$ : $(C - 0 I) = [1&2 2&4]$

Equation: $v_{1} + 2 v_{2} = 0 ⟹ v_{1} = - 2 v_{2}$ . Choose $v_{2} = - 1$ , so $v_{1} = 2$ . Raw: $[2, - 1]^{⊤}$ .

Normalize: $∥ v_{2} ∥ = 2^{2} + (- 1)^{2} = 5$ .

$v_{2} = \frac{1}{5} [2 - 1] \approx [0.894 - 0.447]$

Check orthogonality: $v_{1} \cdot v_{2} = (1) (2) + (2) (- 1) = 2 - 2 = 0$ . ✓ They are perpendicular.

16.7.5 Interpreting the Result

Step 5: Project and analyze.

Variance explained: $\frac{λ _{1}}{λ _{1} + λ _{2}} = \frac{5}{5 + 0} = 1 = 100%$

PC1 captures 100% of the total variance. PC2 captures 0%. This makes perfect sense — the second column was exactly twice the first column. there was only one independent direction of variation to begin with.

Project the centered data onto $v_{1}$ : $z = X_{centered} v_{1}$

Compute each point's projection:

Point 1: $[- 1, - 2] \cdot [1/ 5, 2/ 5]^{⊤} = - 1/ 5 - 4/ 5 = - 5/ 5 = - 5 \approx - 2.236$
Point 2: $[0, 0] \cdot [1/ 5, 2/ 5]^{⊤} = 0$
Point 3: $[1, 2] \cdot [1/ 5, 2/ 5]^{⊤} = 1/ 5 + 4/ 5 = 5/ 5 = 5 \approx 2.236$

$z = - 5 05$

Verify variance matches eigenvalue: $Var (z) = \frac{1}{2} ((- 5)^{2} + 0^{2} + (5)^{2}) = \frac{5 + 0 + 5}{2} = \frac{10}{2} = 5 = λ_{1}$

✓ The variance of projected data equals the eigenvalue, exactly as the theory predicted.

Project onto $v_{2}$ (should give zero variance):

Point 1: $[- 1, - 2] \cdot [2/ 5, - 1/ 5]^{⊤} = - 2/ 5 + 2/ 5 = 0$
Point 2: $[0, 0] \cdot [2/ 5, - 1/ 5]^{⊤} = 0$
Point 3: $[1, 2] \cdot [2/ 5, - 1/ 5]^{⊤} = 2/ 5 - 2/ 5 = 0$

All three points project to 0 along $v_{2}$ . Zero variance in this direction — exactly as $λ_{2} = 0$ predicted. The entire dataset, originally 2D, lives entirely on a 1D line.

Final result: From 2 features to 1 component with zero information loss. PCA detected and exploited the perfect linear dependence $X_{2} = 2 X_{1}$ .

Sense-check: If you were given this dataset and asked "can we reduce from 2D to 1D?" — yes, with 100% fidelity. The single PCA feature $z$ encodes everything the two original features encoded, using half the storage and no redundancy.

Pitfalls in Numerical PCA Problems.

Arithmetic errors in $X^{⊤} X$ . Triple-check every multiplication. A single sign error in step 2 cascades through the entire calculation.
Forgetting the $N - 1$ divisor. Many students compute $C = X^{⊤} X / N$ instead of $/ (N - 1)$ . For small $N$ , this shifts eigenvalues noticeably. The professor uses $N - 1$ .
Confusing eigenvalues and eigenvectors. The eigenvalue is a number; the eigenvector is a direction. "Which direction has maximum variance?" asks for the eigenvector. "How much variance?" asks for the eigenvalue. Read the exam question carefully.
Assuming the eigenvector with $λ_{m a x}$ is always $[1, 1]^{⊤}$ . It points in the direction of maximum spread,. Which for our perfectly correlated data is $[1, 2]^{⊤}$ — a 2:1 slope, not 1:1. You must compute it, not guess.

Recap + Bridge. This example demonstrates PCA in miniature: center the data, compute the covariance. Matrix, find eigenvalues via the characteristic equation, compute eigenvectors, project. Onto the top component. With 100% correlation between the two features, PCA correctly identified that one dimension suffices. The projected variance (5) equals the eigenvalue — a perfect match with theory. The procedure scales to any $D$ , though beyond 3×3 you would use numerical methods, not hand-calculation. Next, we explore the practical trade-offs of PCA: interpretability, information loss, and when the method works versus when it fails.

Real-World & Domain Connection. This hand-calculation reveals the exact same computation done under the hood when you call PCA(n_components=1).fit_transform(X) in scikit-learn. The library uses SVD for numerical stability, but the result is identical. Understanding the manual calculation is not just an academic exercise — it equips you to debug PCA. When it gives unexpected results, to explain your dimensionality reduction choices in a paper or presentation. to reason about what the principal components mean in the context of your data.

16.8 Practical Considerations

(All symbols used in this section are defined in the symbol registry of section 16.5.2.)

Hook. You have just learned the elegant mathematics of PCA. Now the hard question: should you use it? PCA is not always the right tool. It trades interpretability for accuracy. It helps when redundancy is high and hurts when features are already independent. Knowing when to apply PCA — and when to walk away — is as important as knowing how it works.

16.8.1 PCA vs Regression — Two Sides of the Same Coin

PCA and linear regression are mathematically dual perspectives on the same geometric operation: projection onto a line.

	PCA	Regression
Goal	Maximize variance of projected data	Minimize squared error between prediction and target
What is minimized	—	$\sum (y_{i} - y^_{i})^{2}$ (vertical residuals)
What is maximized	$w^{⊤} Cw$ (variance along axis)	—
Supervision	Unsupervised (no target $Y$ )	Supervised (has target $Y$ )
Direction optimization	PCA finds the best axis	Regression axis is determined by $(X, Y)$ relationship

The connection runs deep. If you formulate regression as finding a line through the data that minimizes the perpendicular (not vertical) distance from points. To the line — total least squares — the solution is exactly the first principal component of the joint $(X, Y)$ data. The professor remarked they are "two sides of the same coin" because both boil down to eigendecomposition of a covariance-like matrix.

16.8.2 PCA vs Pairwise Correlation Removal

Why not just compute pairwise correlations and drop one from each pair?

Approach	Cost	Issues
Pairwise correlation check	$(2 D)$ comparisons (4950 for $D = 100$ )	Must decide which feature to keep per pair; iterative and arbitrary
PCA	Single eigendecomposition of $C$	Automatic, handles all correlations simultaneously

First, correlation only measures linear association. If you remove features based on linear correlation. And then use a nonlinear model (neural network, random forest), you are filtering out linear relationships. And then trying to model nonlinear ones — conceptually incoherent.

Second, the combinatorics explode: for 100 features, you examine 4950 correlation pairs, then face a cascade of decisions about. Which feature to keep from each correlated cluster. PCA sidesteps all of this: one pass through the covariance matrix, one eigen decomposition. the math automatically sorts the directions by importance.

16.8.3 The Interpretability Trade-off

The core trade-off: accuracy vs. explainability.

PCA creates principal components $Z_{1}, Z_{2}, \dots$ as weighted sums of all original features: $Z_{1} = a_{1} X_{1} + a_{2} X_{2} + \dots + a_{D} X_{D}$

After PCA, you lose the physical meaning of individual variables.

With original features: "Salary increases by $5,000 per year of experience."
With PCA features: "Salary changes with $Z_{1}$ ," but $Z_{1}$ is a blend of age, experience, education. city — you cannot translate it back to a single actionable business metric.

In business or policy contexts where you must justify decisions ("we recommend increasing the marketing budget because..."), PCA's opaqueness is a problem. The original features, even with a slightly less accurate model, may be preferable because they are explainable.

The Kaggle example shows the trade-off in numbers: PCA-only features gave 88% accuracy (a 20-point jump). the model was a black box. The hybrid approach (PCA + original features) hit 91% — keeping some interpretability while still gaining from the compressed representation.

When to sacrifice interpretability: when prediction accuracy is the sole objective (competitions, automated systems). When to keep original features: when stakeholders need to understand and act on the model's reasoning.

16.8.4 Information Loss

Dimensionality reduction inherently discards information. The question is whether the discarded information matters.

For a $60 \times 60$ grayscale image (3600 pixels), PCA can often reconstruct a recognizable. Version using just 20–24 components — capturing ~99.99% of the variance. The remaining ~3576 components encode only fine textural detail, individual pixel noise, and imperceptible variations. You can throw away over 99% of the dimensions and still recognize the content.

The reconstruction error equals the sum of discarded eigenvalues: $J_{M} = \sum_{j = M + 1}^{D} λ_{j}$ . If these eigenvalues are small (the typical case), the error is negligible. The scree plot — a bar chart of eigenvalues sorted by size — shows you exactly where the drop-off happens. Choose $M$ just after the "elbow" where eigenvalues become tiny and flat.

16.8.5 Orthogonality of Principal Components

A critical property of PCA: all principal components are mutually orthogonal — perpendicular to each other. Because they are orthogonal in the original feature space, the resulting features $Z_{1}, Z_{2}, \dots, Z_{D}$ are linearly uncorrelated.

This is a mathematical guarantee when all features are included in a single PCA run. The eigenvectors of a symmetric matrix (the covariance matrix) are always orthogonal. Therefore:

$Cov (Z_{i}, Z_{j}) = 0$ for all $i \neq = j$ .
This eliminates redundancy between the new features — the exact problem PCA was designed to solve.

Warning: This guarantee only holds when all features enter PCA together. If you pick a subset of features. And do PCA on just those, the resulting components can be correlated with the features you excluded.

16.8.6 When PCA Cannot Reduce Dimensions

PCA is not a magic bullet. It reduces dimensions only when there is redundancy to eliminate.

If your features are all independent and equally informative, the eigenvalues of $C$ will be roughly equal. The scree plot will show no clear drop-off — it will look like a flat line or a gentle slope. In this case, keeping the top $P$ components discards meaningful variation. You started with 100 dimensions, and you still need roughly 100 to capture the variance.

Signs PCA is not helping:

All eigenvalues are similar in magnitude.
The cumulative variance curve climbs linearly, not with an elbow.
Test accuracy drops sharply when you remove components.

In these cases, do not force PCA. Use your original features or consider a different approach (feature selection, nonlinear methods).

16.8.7 PCA, Overfitting, and Regularization

Q&A:

Q: Can PCA lead to overfitting? If so, how do we regularize after PCA?

A: PCA itself does not cause overfitting. It is a data transformation, not a model. It gives you a new input space. After PCA, you train whatever model you choose and apply regularization as usual.

However, there is a subtle interaction: PCA reduces collinearity among features. After PCA, the new features are uncorrelated. This means hard regularization methods like Ridge (L2 penalty, which shrinks correlated coefficients toward zero) or LASSO (L1 penalty,. Which can zero out redundant features) become less effective — because after PCA, the remaining features already capture distinct variance. Their coefficients are genuinely informative and will not be driven to zero.

For neural networks, soft methods like dropout work differently: dropout randomly disables connections during training, forcing the network. To build robust representations without relying on any single pathway. This is independent of whether the inputs came from PCA.

The broader point: regularization is not limited to Ridge, LASSO, and Elastic Net. Different model types have different regularization strategies. Choose based on your model, not based on whether PCA was applied.

16.8.8 Feature Engineering with PCA

Three strategies for using PCA features:

Replace original features entirely — use only the top $P$ PCA components. Best when interpretability is not required and feature count must be minimized.
Keep top $P$ components only — same as above, but with explicit choice of $P$ based on variance threshold.
Hybrid: concatenate PCA features with originals — keep all original features and add PCA components as extra features. This often outperforms either pure approach because the model can use the compressed signal. From PCA while still accessing individual features when needed. The Kaggle team achieved 91% accuracy this way (up from 68% baseline and 88% PCA-only).

The hybrid approach works because PCA components capture the "gist" of the data — the broad correlation patterns —. While the original features preserve granular detail and interpretability. The model learns to blend both sources. :::

16.8.9 Student Questions and Answers

Q&A — Deduplicated. Several students asked about selective PCA, component-feature correlation, eigenvalue attribution, and SVD:

Q: If we have 50 features — 40 correlated. And 10 uncorrelated — should we do PCA only on the 40 correlated ones?

A: No. Run PCA on all 50 features together. The algorithm handles correlations automatically across the full dataset. The uncorrelated features will naturally produce distinct components with moderate eigenvalues. You do not need to pre-sort features into "correlated" and "uncorrelated" buckets — PCA does exactly this for you.

Q: Can a PCA component formed from X1 and X2 be correlated with X5?

A: If you run PCA only on ${X_{1}, X_{2}}$ (a subset), the resulting component could be correlated. With $X_{5}$ — there is no mathematical barrier. But when all features enter PCA together, the algorithm guarantees that every principal component is uncorrelated with every other component. The eigenvectors are orthogonal, so $Cov (Z_{i}, Z_{j}) = 0$ for all $i \neq = j$ . This includes $Z_{1}$ (a blend of all features) and any component representing the direction of $X_{5}$ .

Q: How do we know which original feature an eigenvalue belongs to?

A: An eigenvalue belongs to an eigenvector, not to any single original feature. PCA proves that the variance in the direction of eigenvector $v$ equals its eigenvalue $λ$ . The eigenvector with the largest $λ$ is PC1 — a weighted blend of all original features. The eigenvalue tells you how much variance that blend captures. The components are ordered by $λ$ : PC1 has the largest, PC2 the next, and so on.

Q: Does PCA use only the covariance matrix, or are there alternatives?

A: The next lecture covers SVD — Singular Value Decomposition — which performs PCA without explicitly computing $C$ . SVD decomposes $X$ directly into $U Σ V^{⊤}$ , where the columns of $V$ are the eigenvectors of $X^{⊤} X$ . And the singular values $σ_{k}$ relate to eigenvalues via $λ_{k} = σ_{k}^{2} / (N - 1)$ . Both approaches yield identical results. SVD is preferred computationally for large datasets because it avoids forming $X^{⊤} X$ , which can cause numerical precision loss.

Recap + Bridge. PCA is a powerful tool, not a universal one. It trades interpretability for accuracy, works only when redundancy exists, and guarantees uncorrelated output components. Use it when: features are many and correlated, prediction accuracy matters more than explainability, and you need faster training. Avoid it when: interpretability is critical, features are already independent, or the data has strong nonlinear structure. The next lecture introduces SVD — an alternative computational path to the same solution that handles large-scale and high-dimensional data more efficiently.

Real-World & Domain Connection. The interpretability trade-off is a live issue in regulated industries. In credit scoring, regulators require banks. To explain why a loan was denied — "your PCA component Z3 was below threshold" is not an acceptable reason. Banks use feature selection or inherently interpretable models instead of PCA for these applications. In contrast, quantitative hedge funds use PCA extensively on hundreds of market indicators. To build trading signals — there, prediction accuracy is all that matters and the black-box nature of PCA components is acceptable. The context in which you deploy determines whether PCA's accuracy gain is worth its explainability cost.

Exam Guidance Summary

Exam note: PCA is very likely to appear in the examination this semester. Last semester PCA was assigned as coursework, so it was excluded from the exam. This year it is not an assignment — the professor has explicitly indicated exam questions on PCA are expected.

What to prepare

You may be asked to perform any subset of the full PCA pipeline on a small (2×2 or 3×3) matrix:

Center the data. Given a small matrix $X$ , compute column means and subtract them to get $X_{centered}$ . This tests whether you remember that PCA requires zero-mean data.

Compute the covariance matrix. $C = \frac{1}{N - 1} X_{centered}^{⊤} X_{centered}$ . Remember: $N - 1$ (sample covariance), not $N$ . Double-check every entry of the matrix multiplication.

Find eigenvalues. Solve $det (C - λ I) = 0$ . For a 2×2 matrix, this expands to a quadratic. Expand carefully — the constant terms often cancel (as in section 16.7), simplifying the algebra.

Identify the direction of maximum variance. The eigenvector corresponding to the largest eigenvalue is PC1. Solve $(C - λ_{m a x} I) v = 0$ and normalize the result to a unit vector.

Compute the proportion of variance. $λ_{1} / \sum λ_{i}$ . If the question asks "how much variance is captured by the first $P$ components?", compute $\sum_{k = 1}^{P} λ_{k} / \sum_{k = 1}^{D} λ_{k}$ .

Find the eigenvector for the largest eigenvalue. Covered in section 16.7 with a worked example. Practice solving the homogeneous system $(C - λ I) v = 0$ and normalizing.

Study strategy

Work through the numerical example in section 16.7 at least twice — once with the book open, once from memory. The exam will give you different numbers but the same procedure. Key skills to drill:

Matrix multiplication without mistakes
Determinant expansion of a 2×2 or 3×3 matrix
Solving a homogeneous linear system for eigenvectors
Normalizing vectors to unit length

Different problem types may appear: some may ask for the full pipeline, others may ask only for eigenvalues, eigenvectors. the proportion of variance. Be prepared to execute any individual step on demand.

Common mistakes to avoid in the exam

Using $N$ instead of $N - 1$ in the covariance denominator
Forgetting to center the data before computing $C$
Computing $X X^{⊤}$ (the $N \times N$ Gram matrix) instead of $X^{⊤} X$ (the $D \times D$ covariance)
Reporting an un-normalized eigenvector (the answer must be a unit vector)

Key Industry Applications

Dimensionality Reduction for High-Dimensional Data

PCA compresses datasets with hundreds or thousands of features into a manageable set of principal components. This reduces training time, memory usage, and (by removing noise dimensions) can improve generalization. In genomics, PCA turns ~20,000 gene expression measurements per patient into ~50 components. That capture the biologically meaningful variation, enabling clustering of disease subtypes and drug-response groups.

Feature Engineering in Competitions

Kaggle contestants routinely apply PCA to boost model performance. The professor cited a specific case where PCA-transformed features lifted a regression model. From 68% to 88% accuracy (PCA-only) and to 91% when combined with original features. The typical workflow: (1) build a baseline with raw features, (2) add PCA components. As extra columns, (3) tune the number of components via cross-validation.

Image Compression and Reconstruction

PCA approximates high-resolution images using far fewer dimensions. For a 28×28 MNIST digit (784 pixels), 100 PCA components reconstruct a sharp, recognizable digit; 500 components give near-perfect recovery. JPEG compression uses a related technique (DCT — discrete cosine transform) to achieve similar compression. In both cases, high-frequency detail (fine texture) gets discarded first, while the broad structural information is preserved.

Exploratory Data Analysis

Projecting high-dimensional data onto the first 2 or 3 principal components lets you visualize structure. That is invisible in the original feature space. Scatter plots of PC1 vs. PC2 often reveal natural clusters, outliers, gradients, and class separability. This is frequently the first step in any high-dimensional analysis — before running any model, you "PCA it down. To 2D just to see what is going on."

Noise Filtering

By discarding components with very small eigenvalues, PCA acts as a denoising filter. In signal processing, the low-variance dimensions often capture measurement noise, sensor drift. irrelevant background variation. the high-variance dimensions capture the true signal. This principle is used in EEG signal analysis (isolating brain activity. From muscle artifacts) and in financial data (separating market-wide trends from stock-specific noise).

Preprocessing for Machine Learning Pipelines

Many production ML pipelines include PCA as a standard preprocessing step. Benefits: removes collinearity (which can destabilize linear models), reduces training time (fewer features → faster iteration). sometimes improves generalization (by discarding noise dimensions). Libraries like scikit-learn make this trivial: make_pipeline(PCA(n_components=0.95), LogisticRegression()) applies PCA retaining 95% variance, then trains the classifier on the reduced representation.

Beyond Standard PCA

While not covered in this lecture, the PCA framework extends to:

Kernel PCA: Applies the kernel trick (from SVM theory) to find nonlinear principal components in a high-dimensional feature space.
Sparse PCA: Produces principal components where most weights are exactly zero, improving interpretability.
Robust PCA: Decomposes a data matrix into a low-rank (signal) component and a sparse (outlier) component, making PCA resistant to corrupted observations.

MFML Lecture 16 notes · Principal Component Analysis

Mathematical Foundations for Machine Learning· postgraduate· 2026-07-11

Sections Breakdown

116.1 Introduction and Motivation

What PCA solves: compressing many redundant features into a few uncorrelated directions of maximum variance.

216.2 Two Types of Redundancy

The two kinds of dead weight PCA removes: constant (near-zero-variance) features and highly correlated features.

316.3 Projection and Variance Maximization

How projecting data onto a unit vector works, and why PCA maximizes the variance of the projection.

416.4 Mean Centering

Why subtracting column means puts the data's center at the origin, simplifying the variance and making projections meaningful.

516.5 Mathematical Formulation

The full derivation: variance as a quadratic form, the covariance matrix, and the eigenvalue solution.

616.6 The PCA Algorithm

The five-step PCA procedure and how to choose the number of components from the proportion of variance retained.

716.7 Worked Numerical Example

A complete hand calculation: center, form the covariance matrix, eigen-decompose, and project a 3-point dataset.

816.8 Practical Considerations

PCA versus regression, the interpretability trade-off, information loss, orthogonality, and when PCA cannot reduce dimensions.

9Exam Guidance Summary

What to prepare for an exam question on PCA and the common mistakes to avoid.

10Key Industry Applications

Where PCA is used in practice: genomics, finance, image compression, EDA, and ML pipelines.

Postgraduate students in Mathematical Foundations for Machine Learning

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

PCA and the Motivation for Dimensionality Reduction

Must-know: PCA is a linear method that finds the directions of maximum variance in your data. It rotates them into a new set of uncorrelated axes, keeping only enough to capture most of the information.

$Z = X_{centered} V_{P}, V_{P} = [v_{1}, \dots, v_{P}]$

Top pitfall: Thinking PCA needs features to be independent. It works precisely because features are redundant. If everything is independent, eigenvalues are equal and PCA cannot help.

Self-check: Why does a dataset whose eigenvalues are all equal gain nothing from PCA?

Connects to: Covariance matrix, Eigenvalues, Redundancy: constant and correlated features

Two Types of Redundancy

Must-know: PCA removes two kinds of dead weight. Constant (near-zero-variance) features and highly correlated features that move together.

$Var (X_{j}) = \frac{1}{N - 1} i = 1 \sum N (x_{ij} - x ˉ_{j})^{2}$

Top pitfall: Assuming correlation implies causation. PCA only exploits the linear correlation structure. It treats all features symmetrically.

Self-check: Why does a column stuck at one value get discarded automatically?

Connects to: Variance, Covariance matrix, The PCA algorithm

Projection and Variance Maximization

Must-know: Projecting data onto a unit vector $w$ gives $z_{i} = x_{i}^{⊤} w$ . PCA picks the $w$ that maximizes the variance of these projections.

$z_{i} = x_{i}^{⊤} w, w max Var (Xw) s.t. w^{⊤} w = 1$

Top pitfall: Forgetting the unit-vector constraint. Without $w^{⊤} w = 1$ , you could scale $w$ to make the variance arbitrarily large.

Self-check: What stops the maximum variance from blowing up to infinity?

Connects to: Covariance matrix, Eigenvectors, Mean centering

Mean Centering

Must-know: Subtract each column mean so the data's center of mass sits at the origin. This makes the projected mean zero and keeps projections through the origin meaningful.

$X_{centered} = X - X ˉ, z ˉ = 0$

Top pitfall: Centering the test set with its own mean. Always reuse the training mean, or the PCA projection is inconsistent.

Self-check: Why must $z ˉ = 0$ after centering?

Connects to: Covariance matrix, Projection, The PCA algorithm

The Covariance Matrix and the Eigenvalue Formulation

Must-know: The variance of the projection equals the quadratic form $w^{⊤} Cw$ . Here $C$ is the covariance matrix. The optimal $w$ is an eigenvector of $C$ .

$C = \frac{1}{N - 1} X^{⊤} X, Var (z) = w^{⊤} Cw$

Top pitfall: Using $X X^{⊤}$ (the $N \times N$ Gram matrix) instead of $X^{⊤} X$ (the $D \times D$ covariance). Eigen-decompose $C$ , never $X$ .

Self-check: Why must we eigen-decompose $C$ rather than $X$ ?

Connects to: Eigenvectors, Eigenvalues, Mean centering, Projection

The PCA Algorithm (Five Steps)

Must-know: Center, build the covariance matrix, eigen-decompose it, sort eigenvectors by eigenvalue, then project onto the top $P$ components.

$Z = X_{centered} V_{P}, V_{P} = [v_{1}, \dots, v_{P}]$

Top pitfall: Picking $P$ arbitrarily (for example, two because plots look nice). Justify it with the cumulative variance threshold or the scree-plot elbow.

Self-check: Name the five steps of PCA in order.

Connects to: Covariance matrix, Proportion of variance, Mean centering

Proportion of Variance Explained

Must-know: The variance captured by component $k$ is its eigenvalue $λ_{k}$ . The retained fraction is the sum of the top $P$ eigenvalues over the total.

$\frac{\sum _{k = 1}^{P} λ _{k}}{\sum _{k = 1}^{D} λ _{k}}, Total = k = 1 \sum D λ_{k} = tr (C)$

Top pitfall: Reporting an un-normalized eigenvector. The answer must be a unit vector. Normalize before using it.

Self-check: If $λ_{1} = 40$ and $λ_{2} = 1$ , what fraction of variance does PC1 capture?

Connects to: Eigenvalues, The PCA algorithm, Scree plot

Worked Numerical Example

Must-know: For the 3-point, 2-feature dataset, centering then forming $C$ gives eigenvalues 5 and 0. One component captures 100% of the variance, because the second feature was exactly twice the first.

$det (C - λ I) = 0, λ_{1} = 5, λ_{2} = 0$

Top pitfall: Arithmetic slips in $X^{⊤} X$ . Triple-check every product. One sign error cascades through the whole calculation.

Self-check: Verify by hand that the projected variance equals $λ_{1}$ .

Connects to: Covariance matrix, Eigenvalues, Eigenvectors

PCA vs Regression and the Interpretability Trade-off

Must-know: PCA (unsupervised) maximizes variance of projected data. Regression (supervised) minimizes residual error. PCA trades interpretability for accuracy by blending all features.

$Z_{1} = a_{1} X_{1} + a_{2} X_{2} + \dots + a_{D} X_{D}$

Top pitfall: Trusting a PCA component as if it were a single original feature. $Z_{1}$ is a weighted blend, so you lose the physical meaning of any one variable.

Self-check: When would you keep original features instead of PCA components?

Connects to: The PCA algorithm, Overfitting, Feature engineering

Orthogonality and When PCA Cannot Help

Must-know: Principal components are mutually orthogonal, so the new features are uncorrelated. PCA only helps when real redundancy exists. Equal eigenvalues mean no reduction is possible.

$v_{i}^{⊤} v_{j} = 0 (i \neq = j), Cov (Z_{i}, Z_{j}) = 0$

Top pitfall: Forcing PCA when all eigenvalues are similar. The scree plot is flat, so you still need all original dimensions.

Self-check: What does a flat scree plot tell you about your features?

Connects to: Eigenvectors, Covariance matrix, Proportion of variance

Previous LectureLecture 15 · Constrained Optimization and Duality

Next LectureLecture 17 · Principal Component Analysis — Practical Computation and the Dual Perspective

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Prerequisite Knowledge

Previously Covered in This Subject

Principal Component Analysis

16.1 Introduction and Motivation

16.1.1 What PCA Solves

16.2 Two Types of Redundancy

16.2.1 Constant Features

16.2.2 Correlated Features

16.2.3 Visualizing Correlation with Scatter Plots

16.3 Projection and Variance Maximization

16.3.1 The Unit Vector Direction

16.3.2 The Projection Formula

16.3.3 Maximizing Variance after Projection

16.3.4 Symbol Registry

16.3.5 Student Questions and Answers

16.4 Mean Centering

16.4.1 Why Center the Data

16.4.2 How Mean Centering Works

16.4.3 Student Questions and Answers

16.5 Mathematical Formulation

16.5.1 Variance of Projected Data

16.5.2 Symbol Registry

16.5.3 The Covariance Matrix

16.5.4 Constrained Optimization

16.5.5 Lagrangian Formulation

16.5.6 The Eigenvalue Connection

16.5.7 Variance Equals Eigenvalue

16.5.8 Student Questions and Answers

16.6 The PCA Algorithm

16.6.1 Step-by-Step Procedure

16.6.2 Proportion of Variance Explained

16.6.3 Student Questions and Answers

16.7 Worked Numerical Example

16.7.1 Setting Up the Data

16.7.2 Mean Centering

16.7.3 Computing the Covariance Matrix

16.7.4 Eigen Decomposition

16.7.5 Interpreting the Result

16.8 Practical Considerations

16.8.1 PCA vs Regression — Two Sides of the Same Coin

16.8.2 PCA vs Pairwise Correlation Removal

16.8.3 The Interpretability Trade-off

16.8.4 Information Loss

16.8.5 Orthogonality of Principal Components

16.8.6 When PCA Cannot Reduce Dimensions

16.8.7 PCA, Overfitting, and Regularization

16.8.8 Feature Engineering with PCA

16.8.9 Student Questions and Answers

Exam Guidance Summary

What to prepare

Study strategy

Common mistakes to avoid in the exam

Key Industry Applications

Dimensionality Reduction for High-Dimensional Data

Feature Engineering in Competitions

Image Compression and Reconstruction

Exploratory Data Analysis

Noise Filtering

Preprocessing for Machine Learning Pipelines

Beyond Standard PCA

MFML Lecture 16 notes · Principal Component Analysis

Sections Breakdown

Exam Revision Notes

PCA and the Motivation for Dimensionality Reduction

Two Types of Redundancy

Projection and Variance Maximization

Mean Centering

The Covariance Matrix and the Eigenvalue Formulation

The PCA Algorithm (Five Steps)

Proportion of Variance Explained

Worked Numerical Example

PCA vs Regression and the Interpretability Trade-off

Orthogonality and When PCA Cannot Help

Was this lecture useful?

BitsNotes AI Assistant

Configure AI Chat

Sign in to use AI Chat