Mathematical Foundations for Machine Learning

Principal Component Analysis — Practical Computation and the Dual Perspective

Published: 2026-07-11

Level: Undergraduate

Audience: University students studying the mathematical foundations of machine learning

Prerequisite Knowledge

This lecture builds on the following concepts from earlier lectures. If any feel unfamiliar, review the linked notes before proceeding.

Previously Covered in This Subject

Principal Component Analysis (introduction, projection, covariance, the eigenvalue connection) — covered in Lecture 16
Constrained Optimization and Duality (Lagrangian, Lagrange multipliers, the dual problem) — covered in Lecture 15

PCA — Practical Applications, Worked Computation, and the Dual Perspective

You already know that Principal Component Analysis finds the directions where your data spreads out the most. This chapter turns that idea into a working skill. We revisit the eigenvalue formulation and pin down exactly when PCA succeeds and when it fails. We walk through a full hand computation and prove that maximizing variance equals minimizing reconstruction error. We finish with a trick that makes PCA feasible even when features vastly outnumber samples.

17.1 PCA Recap and Motivation

Suppose I hand you a dataset with 1000 columns. You know in your gut that most of those columns are redundant — maybe only 10 of them carry the real signal. How do you find those 10 mathematically, without guessing? That is what PCA solves: it hunts down the directions in your data where the spread is biggest, ranks them, and lets you keep only the top few.

Think of taking a group photo. Everyone stands in rows, but some people are hidden behind others. If you could walk around and shoot from the side, you would capture everybody — because the people are spread out along a line from front to back. PCA does the same thing: it rotates your "camera" (the coordinate axes) so it looks along the direction where your data spreads out the most. The first new axis — the first principal component — is like that side angle: it captures the biggest differences between data points. The second new axis is perpendicular to the first and captures the next biggest differences. This analogy breaks in one way: PCA does not literally rotate a camera — it finds a new set of perpendicular axes that are linear combinations of the original ones. But the mental picture of "finding the best viewing angle" is exactly right.

17.1.1 Recap: Maximizing Variance via Eigenvectors

PCA is a dimension reduction technique. It finds directions in your data where the spread — measured by variance — is largest. When you project data onto these directions, you get new features that carry most of the information with far fewer dimensions.

The core optimization problem from the previous lecture: find a unit direction vector $w$ (with $∥ w ∥ = 1$ ) that maximizes the variance of the projected data. This is a constrained optimization:

w max w^{⊤} C w subject to w^{⊤} w = 1

Here $C$ is the covariance matrix of your data. The covariance matrix summarizes how every pair of features varies together — each entry $C_{ij}$ is the covariance between feature $i$ and feature $j$ . To solve this constrained problem, you form the Lagrangian:

L (w, λ) = w^{⊤} C w - λ (w^{⊤} w - 1)

The term $λ (w^{⊤} w - 1)$ is the constraint penalty. Lambda ( $λ$ ) is the Lagrange multiplier — a scalar that enforces the unit-length condition on $w$ . Take the partial derivative with respect to $w$ and set it to zero:

\frac{\partial L}{\partial w} = 2 C w - 2 λ w = 0 ⟹ C w = λ w

This is the eigenvalue equation. The direction $w$ that maximizes variance is an eigenvector of the covariance matrix $C$ . The Lagrange multiplier $λ$ that emerged from the constraint becomes the eigenvalue.

Now substitute $C w = λ w$ back into the variance objective:

w^{⊤} C w = w^{⊤} (λ w) = λ (w^{⊤} w) = λ

So $λ$ — the eigenvalue — is literally the amount of variance that direction captures. A bigger eigenvalue means the data spreads further along that eigenvector, which means that direction is more informative.

The recipe, in summary:

Compute the covariance matrix $C = \frac{1}{N - 1} X^{⊤} X$ from your mean-centered data $X$ .
Find its eigenvalues (how much variance each direction captures) and eigenvectors (the directions themselves).
Sort eigenvectors by eigenvalue, descending. Keep the top $k$ .
Project your data: $Z = X W$ , where $W$ has the $k$ chosen eigenvectors as columns.

17.1.2 Symbol Registry — PCA Eigenvalue Formulation

Symbol	Meaning	LaTeX	Type
$X$	Data matrix ( $N$ points, $D$ features)	$X \in R^{N \times D}$	matrix
$C$	Covariance matrix	$C = \frac{1}{N - 1} X^{⊤} X$	$D \times D$ matrix
$w$	Direction vector (principal component)	$w \in R^{D}, ∥ w ∥ = 1$	vector
$λ$	Eigenvalue — variance captured	$λ \in R_{\geq 0}$	scalar
$W$	Matrix of $k$ eigenvectors as columns	$W \in R^{D \times k}$	matrix
$Z$	Transformed data	$Z = XW \in R^{N \times k}$	matrix

A tiny numerical recap. Suppose after mean centering, a 2D dataset has covariance matrix:

C = [1 0.5 0.5 1]

The eigenvalues are $λ_{1} = 1.5$ (larger) and $λ_{2} = 0.5$ (smaller). Their corresponding eigenvectors are:

w^{(1)} = [1/ 2 1/ 2], w^{(2)} = [1/ 2 - 1/ 2]

The first component captures $\frac{1.5}{2.0} = 75%$ of the total variance. The second captures 25%. The eigenvectors are perpendicular — their dot product is $\frac{1}{2} - \frac{1}{2} = 0$ . Sense-check: the eigenvalues (1.5 and 0.5) sum to 2.0, which is the total variance — all information is accounted for. And since $C$ was already built from data with zero mean, the eigenvalue sum naturally equals the total spread.

Scope: When the variance-maximization formulation holds.

Assumption 1: Centered data. The covariance derivation assumes zero-mean data. If your data is not mean-centered, the eigenvectors of $X^{⊤} X$ (without centering) do not point in the directions of maximum variance — they point toward the origin. Always center first.
Assumption 2: Linear relationships. Covariance measures linear co-movement. If your data sits on a circle or a spiral, variance maximization along straight axes will not find a meaningful low-dimensional structure.
Assumption 3: Euclidean geometry. The constraint $w^{⊤} w = 1$ and the objective $w^{⊤} C w$ both use the Euclidean inner product. PCA assumes your data lives in a flat Euclidean space.
What breaks: If features are on vastly different scales (e.g., salary in dollars vs. years of experience), features with larger numeric ranges dominate the covariance matrix. The eigenvectors point toward those high-magnitude features regardless of their actual importance. Standardization (dividing by standard deviation) fixes this.

Visual intuition. Imagine a scatterplot of your data — a cloud of $N$ points in $D$ dimensions. The covariance matrix describes the shape of this cloud as an ellipsoid. The eigenvectors of $C$ are the axes of that ellipsoid: the longest axis is the first principal component, the second-longest perpendicular axis is the second, and so on. The eigenvalue for each axis is the squared length of that axis. When you choose the eigenvectors with the largest eigenvalues, you are literally choosing the longest axes of the data ellipsoid. Projecting onto those axes captures the most "spread." The discarded axes are the short ones — the directions where the cloud is thin.

Now picture rotating a 2D elliptical cloud: the first eigenvector always points along the longest span. If the cloud is nearly a circle, the two eigenvalues are about equal — PCA cannot reduce dimensions well because no single axis captures most of the variance. If the cloud is a thin cigar, one eigenvalue dominates — PCA reduces dimensions brilliantly.

Pitfalls when recapping PCA.

Forgetting to mean-center. If you compute $C = X^{⊤} X$ without subtracting the mean, you get something proportional to the second moment, not the variance. The eigenvectors will be wrong.
Confusing $N$ and $N - 1$ . The sample covariance uses $N - 1$ in the denominator (Bessel's correction). Using $N$ gives the population covariance — a different quantity. The course standard is $N - 1$ .
Thinking eigenvalues are always positive. A covariance matrix is positive semi-definite, so all eigenvalues are $\geq 0$ . If you ever get a negative eigenvalue from a covariance matrix, you made an arithmetic error.
Believing that "more components = always better." Adding components always increases variance captured — the question is whether the gain justifies the extra dimension. A component that captures 0.1% of variance while doubling your feature count is not worth keeping.

Recap: PCA finds the directions (eigenvectors) of maximum variance (eigenvalues) in your data, giving you a ranked list of new axes. Keep the top ones for a compressed, information-rich representation. Bridge: But covariance is a linear measure. In the next section, we ask the natural follow-up: what happens when your data's relationships are not linear?

17.1.3 Why Dimension Reduction Matters

Think of a self-driving car. It processes data from cameras, Lidar, radar, temperature sensors, wind sensors — thousands of features simultaneously. But to detect an obstacle, you do not need all of them at once. Only a subset matters at any moment.

PCA lets you identify those high-information features. If 10 new dimensions out of 1000 capture 95% of the total variance, you can safely discard the other 990. Those 990 contributed only 5% of the information. The self-driving car's perception pipeline can now run faster and with less memory while losing almost no critical signal.

This same logic applies anywhere high-dimensional data meets real-time or storage constraints: genomic sequencing, financial modeling, and recommendation systems all use PCA as their first compression tool.

The idea of dimension reduction also connects deeply to how machine learning models generalize. Fewer features mean fewer parameters to estimate, which means less overfitting and better performance on unseen data — provided you kept the features that carry the real signal.

17.2 When PCA Works and When It Does Not

Why does PCA sometimes work brilliantly — compressing 1000 features into 10 — and sometimes fail completely, unable to reduce even 2 features to 1? The answer comes down to one word: linearity. PCA sees the world through a straight-line lens. If your data's relationships are wiggly, curved, or circular, that lens goes blurry.

Imagine you are trying to describe a winding river on a map. If the river is roughly a straight line, you can summarize it with a single arrow pointing from source to mouth — that is PCA working well: one direction captures almost everything. But if the river snakes in S-curves and loops, a single straight arrow captures very little. You would need many arrows, or better yet, a curved line. PCA is the straight-arrow approach: it works beautifully on straight-line relationships, but fails when the "river" — your data — has bends and curves. The analogy breaks in that PCA is not "one arrow" — it gives you as many orthogonal arrows as you like. But the point holds: when the data is nonlinear, the first few arrows will all capture similar, modest amounts of variance. None of them will dominate.

Symbol registry for this section:

Symbol	Meaning	LaTeX	Type
$C$	Covariance matrix	$C$	$D \times D$ matrix
$λ$	Eigenvalue / ridge penalty	$λ$	scalar
$β$	Regression coefficients	$β$	$D \times 1$ vector
$I$	Identity matrix	$I$	$D \times D$ matrix

17.2.1 Covariance Measures Linear Association

Covariance between two features measures the degree of linear association. It tells you whether two variables move together along a straight-line trend. If $Y$ tends to increase when $X$ increases in a roughly straight-line way, covariance is high (positive or negative depending on direction).

Formally, for mean-centered variables:

Cov (X_{j}, X_{k}) = \frac{1}{N - 1} i = 1 \sum N x_{ij} x_{ik}

This is a sum of products. When $X_{j}$ and $X_{k}$ are both large together (same sign), the product is positive and the covariance grows. When they move in opposite directions (one large positive, one large negative), the product is negative. When there is no consistent pattern, the products cancel out and covariance is near zero.

This is different from "spread" or "relatedness" in a general sense. Covariance specifically captures linear co-movement. If the relationship is curved — say $Y = X^{2}$ with $X$ symmetric around zero — then $X$ and $Y$ are perfectly related (knowing $X$ tells you $Y$ exactly), but the covariance is zero. The positive products from $+ X$ values cancel the negative products from $- X$ values.

Comparison: Linear vs. Nonlinear Association

Property	Linear Association (Covariance/Correlation)	Nonlinear Association
Definition	Variables move together along a straight line	Variables are related but the relationship curves or bends
Measured by	Covariance, Pearson correlation ( $r$ )	No single summary statistic exists
Example	House size predicts price (roughly straight line)	Speed of a car predicts fuel efficiency (U-shaped curve)
PCA behavior	Works well — dominant eigenvector captures spread	Fails — eigenvalues are similar, no clear dominant axis
Covariance value for $Y = X^{2}$ (symmetric X)	Zero (cancellation of signs)	Relationship exists but covariance misses it entirely
Approach needed	Linear feature reduction (PCA)	Nonlinear methods (t-SNE, autoencoders, kernel PCA)

When to pick which: if a scatterplot of your features shows roughly straight-line trends, PCA is appropriate. If the scatterplot curves, bends, or forms clusters, reach for a nonlinear method instead.

Q: Is PCA applicable to independent features too, or only dependent ones?

A: PCA applies when features are dependent (correlated). With independent features, you can directly compute $β = (X^{⊤} X)^{- 1} X^{⊤} Y$ without needing regularization tricks like adding $λ I$ . The whole motivation for ridge regression came from correlated features. When features are correlated, $X^{⊤} X$ becomes non-invertible — its determinant is zero because columns are linear combinations of each other. Adding a small $λ$ to the diagonal breaks the linear dependency and makes inversion possible. The ridge penalty $λ \sum β_{j}^{2}$ directly addresses the multicollinearity that PCA also solves.

Concrete example: covariance on linear vs. quadratic data.

Consider three data points with two features. First, a linear case:

$X_{1}$	$X_{2} = 2 X_{1} + 1$
1	3
2	5
3	7

Mean-center: $X ˉ_{1} = 2$ , $X ˉ_{2} = 5$ . Centered values:

$X_{1} - X ˉ_{1}$	$X_{2} - X ˉ_{2}$
$- 1$	$- 2$
0	0
1	2

Covariance: $\frac{( - 1 ) ( - 2 ) + ( 0 ) ( 0 ) + ( 1 ) ( 2 )}{2} = \frac{4}{2} = 2$ . High covariance — strong linear association. PCA would find one dominant direction.

Now a quadratic case:

$X_{1}$	$X_{2} = X_{1}^{2}$
$- 2$	4
0	0
2	4

Mean-center: $X ˉ_{1} = 0$ , $X ˉ_{2} = 8/3 \approx 2.67$ . Centered values:

$X_{1} - X ˉ_{1}$	$X_{2} - X ˉ_{2}$
$- 2$	1.33
0	$- 2.67$
2	1.33

Covariance: $\frac{( - 2 ) ( 1.33 ) + ( 0 ) ( - 2.67 ) + ( 2 ) ( 1.33 )}{2} = \frac{- 2.66 + 0 + 2.66}{2} = 0$ .

Covariance is zero even though $X_{2}$ is perfectly determined by $X_{1}$ . The ± signs cancel. PCA sees no linear association and cannot reduce dimensions. Sense-check: with perfect curvature and symmetric X, PCA should fail — and indeed the covariance being zero confirms the failure.

17.2.2 PCA Fails on Nonlinear Data

Scope: When PCA breaks down.

PCA depends on covariance, and covariance is a linear measure. So PCA only succeeds when your data's relationships are roughly linear.

When PCA works brilliantly: Data is highly collinear — features lie near a straight line or a low-dimensional hyperplane. One direction captures nearly all variance, so you can reduce dimensions aggressively and lose almost nothing.
When PCA fails: Data is spread out in a circle, a spiral, or on a curved manifold. There is no single straight-line direction that captures most of the variance. Rotate the axes and every direction explains roughly the same amount. PCA cannot meaningfully reduce dimensions because all eigenvalues are similar in magnitude.
Formal condition: PCA is optimal when the data lies on (or near) a linear subspace of $R^{D}$ . When the data lies on a nonlinear manifold (a curved surface in the high-D space), PCA cannot capture the structure. It cuts through the manifold rather than following it.
What breaks: If you force PCA on nonlinear data and keep components anyway, you get features that are linear combinations of originals — you lose the nonlinear structure that contained the real signal. A neural network fed these features can never recover what PCA destroyed.

Visual intuition. Picture a scatterplot with axes $X_{1}$ (horizontal) and $X_{2}$ (vertical).

Linear case (cigar shape): Points form a diagonal streak from bottom-left to top-right. The first eigenvector arrow runs right through the middle of the streak, and the data spreads widely along it. The second eigenvector is a short perpendicular arrow — most points huddle close to the streak. The two eigenvalues are very different (e.g., 10 vs. 0.2). PCA works: keep the first component, discard the second.
Circular case (ring shape): Points form a roughly circular cloud. Draw any line through the center — the spread along that line is about the same as along any other. The two eigenvalues are nearly equal (e.g., 2 vs. 1.9). PCA cannot reduce dimensions because no single axis dominates. Keep one and you lose about 50% of the information.
Spiral case: Points trace a spiral out from the center. The data lives on a 1D curve (you only need one coordinate — the angle — to describe every point), but that curve is nonlinear. PCA's straight axes cut across the spiral, each capturing modest variance. The eigenvalues will all be similar even though the intrinsic dimensionality is 1. This is a critical lesson: low intrinsic dimensionality does not guarantee PCA success — the manifold must be linear.

Now imagine rotating the "camera" — the coordinate axes. In the linear case, rotating to align with the cigar shape makes one axis suddenly capture nearly all the variance. In the circular case, no matter how you rotate, each axis still captures roughly half. This is why PCA's eigenvalue spread is the diagnostic: if eigenvalues drop off sharply, linear structure exists. If they stay flat, either the data is truly high-dimensional or the structure is nonlinear.

Pitfalls when assessing PCA applicability.

Confusing correlation with causation. High covariance does not mean one variable causes the other — only that they move together linearly. PCA does not care about causation.
Assuming low eigenvalue spread = no structure. Data may have rich nonlinear structure (clusters, spirals, manifolds) that PCA eigenvalues cannot detect. A flat eigenvalue spectrum means "no strong linear directions," not "no structure."
Applying PCA before checking for linearity. Always scatterplot your data or check pairwise correlations first. If features are weakly correlated, PCA will not help and may hurt.
Confusing "PCA fails" with "the data has high intrinsic dimensionality." A spiral in 2D has intrinsic dimensionality 1 but PCA sees it as 2D. The failure is due to nonlinearity, not due to the data truly needing 2 dimensions.

17.2.3 Beyond PCA: t-SNE and Encoder-Decoder

Two important nonlinear alternatives:

t-SNE (t-distributed Stochastic Neighbor Embedding): Preserves local neighborhood structure — points close in the original high-dimensional space stay close in the low-dimensional embedding. It models similarity with probability distributions over pairs of points. Crucially, t-SNE is not a linear transformation: you cannot write down a matrix $W$ as with PCA. It is an iterative optimization. t-SNE excels at visualization but does not produce a reusable projection matrix for new data points.
Encoder-decoder architectures: Neural networks that compress high-dimensional data through a bottleneck layer. The encoder maps $x \to z$ (high-D to low-D), and the decoder maps $z \to x ~$ (low-D back to high-D). The network is trained to minimize reconstruction error $∥ x - x ~ ∥^{2}$ . Because neural networks have nonlinear activation functions (ReLU, sigmoid, tanh), they can learn curved manifolds. The encoder is the nonlinear equivalent of PCA's projection matrix $W$ .

These are nonlinear feature reduction methods. PCA is linear feature reduction. The distinction is not just academic — it determines which preprocessing pipeline to use.

Q: Do we have a metric to measure nonlinear association, like covariance measures linear association?

A: There is no single concrete summary statistic for nonlinear association analogous to covariance. Mutual information from information theory comes closest — it measures how much knowing one variable reduces uncertainty about another, regardless of whether the relationship is linear. But it requires estimating probability distributions and is significantly more complex than covariance. In neural networks, encoder architectures implicitly learn nonlinear feature transformations. Take an encoder with 4 input neurons, a hidden layer of 3, another of 2, and a single bottleneck neuron: it maps 4 features down to 1 through nonlinear transformations. The network's weights encode the nonlinear association. But there is no single scalar statistic like correlation for nonlinear dependence.

Real-world and domain connection. t-SNE is the go-to tool in genomics and single-cell RNA sequencing for visualizing cell types in 2D. Researchers regularly project thousands of gene expression measurements into two t-SNE coordinates to discover cell subpopulations. In computer vision, autoencoders (encoder-decoder networks) were the precursor to modern generative models like VAEs and diffusion models; the latent space they learn is a nonlinear generalization of PCA's principal subspace. The encoder-decoder perspective also connects PCA to modern deep learning. PCA is a linear autoencoder. If you replace the neural network's activation functions with the identity, the optimal weights are exactly the PCA eigenvectors. This means that every deep autoencoder you train contains PCA as a special, simplified case.

Recap: PCA works when covariance captures the data's structure — which means linear relationships. PCA fails on curves, circles, and spirals where covariance is near zero despite strong relationships. For nonlinear data, reach for t-SNE, autoencoders, or kernel PCA. Bridge: But when the data is linear, PCA gives you something powerful beyond just compression: it removes multicollinearity. The new features are uncorrelated by construction. This is the topic of the next section — and it connects PCA directly to the origins of ridge regression.

Before moving on, it helps to fix in mind what an examiner will probe about this boundary between linear and nonlinear structure.

Exam note: Expect conceptual questions about when PCA works vs. when it fails, and what alternatives exist. Know that covariance → zero does not imply independence — it only implies no linear association. Be able to contrast linear PCA with nonlinear t-SNE/autoencoders on dimensions of: what they optimize, whether they produce a projection matrix, and what data structures they can capture.

17.3 The Practical PCA Pipeline

You know PCA finds directions of maximum variance. But knowing the math is one thing. Knowing when to use PCA and when to skip it, plus how to fit it into a real ML pipeline — that separates a practitioner from someone who only knows the formula.

This section is about the practical rules: what PCA fixes, what it breaks, and how to choose how many components to keep.

Picture a toolbox. Linear regression is a screwdriver — it works on straight, rigid fasteners. A neural network is a power drill with adjustable torque — it handles curves and angles. PCA is like a filter that straightens everything before it reaches the tool. Feed straightened parts to a screwdriver, and the screwdriver works even better — fewer stripped screws, faster assembly. But feed straightened parts to a power drill, and you have wasted the drill's ability to handle angles. The drill was going to create its own angles anyway. That is PCA + linear regression vs. PCA + neural networks in a nutshell.

Symbol registry for this section:

Symbol	Meaning	LaTeX	Type
$Z_{j}$	$j$ -th principal component / transformed feature	$Z_{j}$	scalar column in $Z$
$a_{ij}$	Coefficient in linear combination for $Z_{i}$	$a_{ij}$	scalar (element of $W$ )
$λ$	Eigenvalue / ridge penalty parameter	$λ$	scalar
$β$	Regression coefficient vector	$β$	$D \times 1$ vector
$W$	Matrix of eigenvectors (principal directions)	$W$	$D \times k$ matrix
$Z$	Transformed data matrix	$Z = XW$	$N \times k$ matrix

17.3.1 PCA Removes Multicollinearity

A critical property of PCA: the new features $Z_{1}, Z_{2}, \dots, Z_{k}$ are uncorrelated with each other. This follows directly from eigenvector orthogonality — for any two eigenvectors $w^{(i)}$ and $w^{(j)}$ ( $i \neq = j$ ), their dot product is zero:

w^{(i)} \cdot w^{(j)} = 0

Since each transformed feature is $Z_{j} = X w^{(j)}$ (the data projected onto eigenvector $j$ ), and the projection directions are perpendicular, the resulting features have zero linear correlation. PCA transforms correlated inputs into uncorrelated outputs by construction.

Why does this matter? Multicollinearity — the presence of correlated input features — causes three problems in ML algorithms:

Non-invertible $X^{⊤} X$ . In linear regression, the closed-form solution is $β = (X^{⊤} X)^{- 1} X^{⊤} Y$ . When columns of $X$ are linear combinations of each other, $X^{⊤} X$ has determinant zero. The inverse does not exist. The solution "blows up" — coefficients become infinite.
Overfitting from redundancy. The model learns the same signal multiple times from correlated features, over-weighting that information.
Unstable coefficient estimates. Small changes in the data cause large swings in the estimated $β$ values. You cannot trust the interpretation of individual coefficients.

PCA solves all three by replacing $X$ with $Z = X W$ , where $Z^{⊤} Z$ is diagonal (all off-diagonal covariances are zero).

Historical note: The origin of ridge regression. Before modern ML, statisticians in the 1960s-70s faced this exact problem — they could not invert $X^{⊤} X$ when columns were dependent. Their solution was elegantly simple: add a small number $λ$ to every diagonal element of $X^{⊤} X$ :

β_{ridge} = (X^{⊤} X + λ I)^{- 1} X^{⊤} Y

If column $X_{2}$ is a near-perfect copy of $X_{1}$ ( $X_{2} \approx 2 X_{1}$ ), then $X^{⊤} X$ is nearly singular. Adding $λ I$ nudges every diagonal entry slightly upward, breaking the exact linear dependency. The matrix becomes invertible. The term $λ \sum β_{j}^{2}$ in the ridge objective is the penalty that creates this diagonal shift. PCA and ridge regression attack the same problem from different angles. PCA decorrelates the features before regression; ridge regression modifies the objective to tolerate correlation.

17.3.2 PCA + Linear Regression Works; PCA + Neural Networks Does Not

Each PCA feature $Z_{j}$ is a linear combination of the original inputs:

Z_{1} Z_{2} Z_{3} = a_{11} X_{1} + a_{12} X_{2} + a_{13} X_{3} = a_{21} X_{1} + a_{22} X_{2} + a_{23} X_{3} = a_{31} X_{1} + a_{32} X_{2} + a_{33} X_{3}

The coefficients $a_{ij}$ come from the eigenvectors. Column $j$ of $W$ contains $[a_{1 j}, a_{2 j}, a_{3 j}]^{⊤}$ . In matrix form: $Z = X W$ . This is a linear transformation — every new feature is a weighted sum of the originals, with no nonlinearity involved.

The practical consequence:

PCA + linear regression makes sense. Linear regression already models $Y$ as a linear combination of inputs: $Y^= β_{0} + β_{1} X_{1} + \dots + β_{D} X_{D}$ . Feeding it $Z = X W$ just means the model is learning $Y^= β_{0} + β_{1} (w_{1}^{⊤} x) + \dots + β_{k} (w_{k}^{⊤} x)$ . This is still a linear function of the original $X$ — you are simply working in a rotated, decorrelated coordinate system. The model's representational power is unchanged, but training becomes better-behaved.
PCA + neural network does NOT make sense. A neural network has activation functions ( $σ$ , ReLU, tanh) that introduce nonlinearity. The whole point of a deep network is to learn nonlinear feature combinations in its hidden layers. If you first strip the data down to linear combinations via PCA, you destroy the nonlinear structure the network was designed to exploit. You are feeding the network data that has already been "flattened."

Q: PCA removes linear association and also captures linear association — how do these two statements go together? Aren't they contradictory?

A: They describe different things. PCA is a linear transformation ( $Z = X W$ ). So the new features it produces are linear combinations of the originals — that is the "captures linear association" part. At the same time, the new features are mutually orthogonal — their dot product is zero — so they have no linear association with each other. That is the "removes linear association" part. You start with correlated $X$ , you end with uncorrelated $Z$ . The transformation itself is linear; its output is decorrelated. No contradiction.

Comparison: PCA as preprocessing — when to use it.

Model type	Use PCA before it?	Why / Why not
Linear regression	Yes	Both are linear. PCA decorrelates inputs, making $(X^{⊤} X)^{- 1}$ stable.
Logistic regression	Yes	Same logic — the linear decision boundary benefits from decorrelated inputs.
Ridge / Lasso regression	Sometimes	Ridge already handles multicollinearity via the penalty. PCA can still reduce dimension.
SVM with linear kernel	Yes	Linear SVM works with inner products; PCA just rotates coordinates.
SVM with RBF kernel	No	RBF kernel captures nonlinear structure — PCA would strip it.
Random Forest / XGBoost	No	Tree-based models split on individual features. They do not benefit from linear combinations and can exploit nonlinear associations.
Neural network	No	Hidden layers do implicit nonlinear feature engineering. PCA removes the nonlinearity the network needs.

When to pick which: use PCA before linear models to improve conditioning and reduce dimension. Skip PCA before nonlinear models — they either do not benefit (trees) or are actively harmed (neural networks).

17.3.3 The Standard Industry Pipeline — A Procedural Workflow

Purpose: When you have thousands of input features, applying a single dimensionality reduction method is rarely optimal. Linear redundancy and nonlinear structure are different problems that need different tools. The standard pipeline chains them: PCA strips linear redundancy first, then a nonlinear method captures what remains.

Inputs: Original data matrix $X \in R^{N \times D}$ with $D$ potentially very large (hundreds to thousands of features).

Outputs: A reduced feature matrix with far fewer columns ( $k ≪ D$ ), carrying nearly all the predictive signal from the original data.

Steps:

Apply PCA first to remove multicollinearity and redundant linear features. This is a safe, deterministic step — you keep components capturing, say, 95-97% of variance. From 1000 features, you might drop to 70. The output $Z_{pca}$ has uncorrelated columns.
Then apply t-SNE or an encoder-decoder to the PCA-reduced features. This extracts nonlinear structure. Since the input to this step is already smaller and better-conditioned (70 features instead of 1000), the nonlinear method converges faster and produces more stable embeddings. You might end up with 10 features.
Train your final model (regression, classifier, whatever) on those 10 features.

Trace: Oil extraction with 273 features.

Step 1 — PCA on 273 geological and chemical features → keep components capturing 97% variance → ~75 features.

Step 2 — No further nonlinear reduction was needed in this case (the relationships were predominantly linear).

Step 3 — Train regression model on 75 PCA features.

Result: R² improved from 58% to 60%. On a per-well cost basis, a 2% improvement in yield prediction translates to millions saved across operations. Sense-check: 273→75 is about a 3.6× reduction in features. The 2% gain is modest but economically massive at scale. If the data had strong nonlinear structure, an intermediate t-SNE or autoencoder step would be inserted between Steps 1 and 3.

When this pipeline is appropriate — and when it is not.

Good fit: High-dimensional structured data with both linear redundancy (correlated sensor readings, overlapping financial indicators) and potential nonlinear patterns. The two-stage approach respects both.
Poor fit: Data that is already low-dimensional ( $D < 20$ ). The overhead of the pipeline outweighs the benefits. Just train directly.
Poor fit: Purely linear problems. Skip the nonlinear stage.
Poor fit: Purely nonlinear problems where linear preprocessing destroys signal (e.g., image pixels for a CNN). Skip PCA entirely.

Real-world and domain connection. Oil extraction companies like Halliburton and Schlumberger use this approach. One project involved 273 geological and chemical features to predict oil yield per well. PCA reduced them to ~75 features capturing 97% of variance. The R² improved from 58% to 60% — a 2% gain worth millions of dollars in extraction optimization. Even a 0.5% accuracy improvement translates to massive savings when each well costs millions. This pattern — PCA for linear compression, then nonlinear methods or direct modeling — is standard across three domains:

Finance: hundreds of economic indicators → PCA → trading signals.
Genomics: thousands of gene expressions → PCA → clustering.
Recommendation systems: millions of user-item interactions → PCA → collaborative filtering.

17.3.4 The Pitfall of Manual Correlation-Based Removal

A common beginner mistake: looking at a correlation heatmap and manually dropping correlated features. You see that $X_{1}$ and $X_{2}$ are correlated at 0.9, so you drop one of them. This feels logical — keep only one from each correlated pair. It is dangerous.

Worked example of the trap. Suppose your correlation matrix looks like this:

	$X_{1}$	$X_{2}$	$X_{3}$
$X_{1}$	1.0	0.9	0.8
$X_{2}$	0.9	1.0	0.7
$X_{3}$	0.8	0.7	1.0

$X_{1}$ and $X_{2}$ are correlated at 0.9 — very high. So you test three 2-feature models to decide which to drop:

Model with ${X_{1}, X_{3}}$ → 90% accuracy
Model with ${X_{2}, X_{3}}$ → 70% accuracy (first attempt)
Model with ${X_{2}, X_{3}}$ → 75% accuracy (retry)

You pick the 90% model and discard $X_{2}$ . This is risky.

The problem: correlation (like covariance) only captures linear dependency. $X_{2}$ might have strong nonlinear predictive power for your target $Y$ — a quadratic relationship, an interaction effect, or a threshold effect. By dropping it based on its linear correlation with $X_{1}$ , you permanently lose that nonlinear signal.

Models like random forests, XGBoost, SVMs with nonlinear kernels, and neural networks excel at exploiting exactly this kind of nonlinear association. A random forest trained on all three features might have extracted key splits from $X_{2}$ that $X_{1}$ alone could never provide, despite their 0.9 correlation.

Sense-check: A correlation of 0.9 means $X_{2} \approx 0.9 \cdot X_{1} + noise$ . That noise term might be the key predictor of $Y$ . By dropping $X_{2}$ , you throw away the noise — and the signal it carries.

Pitfalls summary for this section.

Dropping features based on correlation alone. Correlation captures only linear dependence. Nonlinear predictive power is invisible to the correlation coefficient.
Applying PCA before every model without thinking. PCA is a linear preprocessing step. It helps linear models, hurts or does nothing for nonlinear ones. Know your downstream model before you preprocess.
Assuming PCA always reduces overfitting. PCA reduces dimension and removes collinearity, which can reduce overfitting. But if you keep too many components chasing a high variance threshold, you retain noise and overfit anyway. The variance threshold is a guideline, not a guarantee.
Using the pipeline as a black box. Automating PCA → t-SNE → model without checking whether each step actually helps your specific data. Not every dataset benefits from every stage.

17.3.5 Choosing the Variance Threshold

How many principal components should you keep? The answer depends on the cumulative proportion of variance explained.

Total variance equals the sum of all eigenvalues:

Total Variance = i = 1 \sum D λ_{i}

The proportion captured by the first $k$ components:

Proportion_{k} = \frac{\sum _{i = 1}^{k} λ _{i}}{\sum _{i = 1}^{D} λ _{i}}

Since eigenvalues are sorted in descending order ( $λ_{1} \geq λ_{2} \geq \dots \geq λ_{D} \geq 0$ ), taking the first $k$ components always gives the maximum possible variance for a $k$ -dimensional projection.

Common thresholds and when to use them:

Threshold	When to use
80-90%	Exploratory analysis, visualization, or when model simplicity is the priority
95%	Standard default for most ML pipelines — good balance of compression and retention
97-99%	High-stakes problems (oil extraction, medical diagnosis, financial trading) where losing any signal is costly
99.9%+	Essentially no dimension reduction — you kept almost everything. Usually not worth it

Visual intuition. Imagine a bar chart. The horizontal axis lists the eigenvalues $λ_{1}, λ_{2}, \dots, λ_{D}$ sorted from largest to smallest. The vertical axis is the eigenvalue value. Typically, the first few bars are tall (large eigenvalues), and then the bars drop sharply into a long, flat tail of near-zero values. The "elbow" — where the bars stop dropping steeply and flatten out — is a natural cutoff. The cumulative proportion plot (a rising curve from 0% to 100%) shows how quickly you accumulate variance: steep at first, then a slow crawl. The 95% threshold is a horizontal line crossing this curve; the $k$ where they intersect is your answer.

If the bars never drop — all eigenvalues are similar — you have a problem. Either the data is truly high-dimensional (no compression possible), or the relationships are nonlinear (PCA cannot find them). Either way, forcing a threshold will discard real signal.

Q: If we use 97% variance retention, aren't we moving toward overfitting by keeping near-noise features?

A: Reducing from 100 features to 20 that capture 97% variance is still significant dimension reduction. You discarded 80 features. Whether 97% is too aggressive depends on context. In high-stakes problems — predicting oil well yield, detecting cancer from medical images, forecasting market crashes — preserving every bit of information is critical. A discarded "noise" component might carry a subtle but real signal. But you must always validate: a higher variance threshold means more features, which means more parameters to estimate, which can lead to overfitting if your sample size is modest. The professor's advice: spend more time on exploratory data analysis and feature understanding than on hyperparameter tuning. Feature engineering (including PCA threshold choice) deserves more attention than model selection.

Q: What if multiple different combinations of components all cross the 95% threshold? Which do we choose?

A: Choose the one with the fewest components. The goal is dimension reduction — fewer dimensions that still capture adequate variance is strictly better. Since eigenvalues are sorted in descending order, the first $k$ components always capture the most variance for any given $k$ . Among all subsets of size $k$ , the top- $k$ by eigenvalue is optimal. So if 3 components capture 95% and 5 also capture 95%, pick 3. You never need to search over combinations — the sorted ordering does the work for you.

Exam note: Expect problems where you are given a list of eigenvalues and asked "How many components are needed to capture 90% of variance?" Follow this procedure:

Sum all eigenvalues for total variance.
Compute each eigenvalue's proportion: $λ_{i} / \sum λ$ .
Compute cumulative proportions: $Cum_{k} = \sum_{i = 1}^{k} λ_{i} / \sum λ$ .
Find the smallest $k$ where $Cum_{k}$ exceeds the threshold.

With the theory of component selection in hand, the payoff of PCA as preprocessing comes into full view.

Recap: PCA removes multicollinearity by producing orthogonal features, making it a natural preprocessing step for linear models. It actively harms nonlinear models by stripping the structure those models exploit. The industry pipeline chains PCA (linear compression) with nonlinear methods for maximum reduction. The variance threshold — typically 90-95% — determines how many components to keep; eigenvalues sorted descending guarantee the first $k$ is always optimal. Bridge: The next section works through the full PCA computation by hand on a concrete dataset — mean centering, covariance, eigenvalues, eigenvectors, projection, and variance proportions. Every step you just learned about in theory, now with real numbers.

17.4 Worked PCA Computation

You have seen the theory — maximizing variance, eigenvectors, eigenvalues. But theory only sticks when you work through a real computation. This section walks through PCA on a tiny dataset, by hand, so you see exactly what happens at every step. No black boxes, no .fit_transform() shortcuts. After this, you will be able to do a full PCA computation on any small matrix, which is exactly what exams ask for.

Think of PCA as a recipe with six ingredients: your data, the mean of each feature, the covariance matrix, its eigenvalues, its eigenvectors, and finally the projected data. Each step transforms the previous ingredient into the next. If you have ever followed a baking recipe — flour → dough → shaped loaves → baked bread — this is the same idea. Start with raw data, end with compressed features. The analogy breaks at one point: unlike baking, you can (and should) verify every intermediate step. If your eigenvectors are not orthogonal, you made a mistake at the eigenvalue step. If your variance proportions do not sum to 100%, you made a mistake at the projection step. This recipe has built-in self-checks.

Here is a complete numerical walkthrough of PCA on a small dataset. Follow every step.

Data:

$X_{1}$	$X_{2}$
$- 1$	$- 1$
0	1
1	0

Three data points ( $N = 3$ ), two features ( $D = 2$ ).

Purpose: Given a dataset matrix $X$ , produce a reduced representation $Z$ with fewer columns, where each new column is a linear combination of the originals and the new columns are uncorrelated.

Inputs: Data matrix $X \in R^{N \times D}$ . Here $N = 3, D = 2$ .

Outputs: Transformed data $Z = X W \in R^{N \times k}$ where $k \leq D$ . Eigenvalues $λ_{i}$ indicate variance captured by each component. Eigenvectors $w^{(i)}$ are the projection directions.

17.4.1 Step 1: Mean Centering

Rationale: PCA is about variance — how data spreads around its mean. If you do not center the data, the eigenvectors point toward the origin rather than along directions of maximum spread. Centering makes the data zero-mean, so the covariance matrix captures spread, not location.

Compute the mean of each column:

X ˉ_{1} = \frac{- 1 + 0 + 1}{3} = 0, X ˉ_{2} = \frac{- 1 + 1 + 0}{3} = 0

Both means are already zero. So subtracting zero leaves the data unchanged. The mean-centered data matrix is:

X = - 1 01 - 1 10

The professor deliberately chose data with zero mean to simplify. In real problems, you subtract the mean from every column.

Exam note: Mean centering is Step 1 in every PCA problem. Forgetting it loses marks. Always compute $X ˉ_{j}$ and subtract it from each value in column $j$ .

17.4.2 Step 2: Covariance Matrix

Rationale: The covariance matrix summarizes pairwise linear relationships between features. Its eigenvectors will be the PCA directions, and its eigenvalues will be the variance captured by each direction.

After mean centering, the covariance matrix is:

C = \frac{1}{N - 1} X^{⊤} X

Use $N - 1$ (sample covariance), not $N$ . This is a common grading point. The factor $N - 1$ (Bessel's correction) gives an unbiased estimate of the population covariance from a sample.

First compute $X^{⊤} X$ :

X^{⊤} = [- 1 - 1 0110], X = - 1 01 - 1 10

X^{⊤} X = [(- 1) (- 1) + (0) (0) + (1) (1) (- 1) (- 1) + (1) (0) + (0) (1) (- 1) (- 1) + (0) (1) + (1) (0) (- 1) (- 1) + (1) (1) + (0) (0)]

= [1 + 0 + 1 1 + 0 + 0 1 + 0 + 0 1 + 1 + 0] = [2112]

Now divide by $N - 1 = 2$ :

C = \frac{1}{2} [2112] = [1 0.5 0.5 1]

The covariance matrix is symmetric — the off-diagonal elements (0.5) are equal. This is always true because $Cov (X_{1}, X_{2}) = Cov (X_{2}, X_{1})$ .

Exam note: In a previous exam, the solution manual used $1/ N$ instead of $1/ (N - 1)$ . Students who correctly used $N - 1$ (as taught) got marked down because their answers did not match the manual. This was later corrected. The lesson: for sample data, use $N - 1$ . Two conventions exist — $N$ for population, $N - 1$ for sample. The course standard is $N - 1$ .

Q: Is $N$ the number of features or the number of data points?

A: $N$ is the number of data points (rows). $D$ is the number of features (columns). Standard convention: $X \in R^{N \times D}$ .

17.4.3 Symbol Registry — Worked PCA Computation

Symbol	Meaning	LaTeX	Type / Domain
$X$	Mean-centered data	$X \in R^{3 \times 2}$	matrix
$C$	Covariance matrix	$C = \frac{1}{N - 1} X^{⊤} X$	$2 \times 2$ matrix
$λ_{1}, λ_{2}$	Eigenvalues	$λ_{i} \in R_{\geq 0}$	scalars
$w^{(1)}, w^{(2)}$	Eigenvectors	$w^{(i)} \in R^{2}, ∥ w^{(i)} ∥ = 1$	unit vectors
$Z$	Projected data	$Z = XW$	$3 \times 2$ matrix

17.4.4 Step 3: Finding Eigenvalues

Rationale: Eigenvalues tell you how much variance each principal component captures. You solve the characteristic equation to find them.

Solve the characteristic equation:

det (C - λ I) = 0

det (1 - λ 0.5 0.5 1 - λ) = 0

(1 - λ)^{2} - (0.5) (0.5) = 0

(1 - λ)^{2} = \frac{1}{4}

1 - λ = \pm \frac{1}{2}

Case 1: $1 - λ = + \frac{1}{2}$ → $λ = 1 - \frac{1}{2} = \frac{1}{2} = 0.5$

Case 2: $1 - λ = - \frac{1}{2}$ → $λ = 1 + \frac{1}{2} = \frac{3}{2} = 1.5$

So the eigenvalues are:

λ_{1} = 1.5 (larger — captures more variance), λ_{2} = 0.5 (smaller)

17.4.5 Step 4: Finding Eigenvectors

Rationale: Eigenvectors are the PCA directions. Each eigenvector is a unit vector pointing in one principal direction. You find them by plugging each eigenvalue back into $(C - λ I) w = 0$ and solving.

For each eigenvalue, solve $(C - λ I) w = 0$ .

For $λ_{1} = 1.5$ :

[1 - 1.5 0.5 0.5 1 - 1.5] [x y] = [- 0.5 0.5 0.5 - 0.5] [x y] = [00]

From the first row: $- 0.5 x + 0.5 y = 0$ → $x = y$ .

The direction is $(1, 1)$ . Normalize to unit length:

∥ w^{(1)} ∥ = 1^{2} + 1^{2} = 2

w^{(1)} = [\frac{1}{2} \frac{1}{2}]

You always normalize eigenvectors because PCA uses unit directions. The $w^{⊤} w = 1$ constraint in the Lagrangian demands it. An unnormalized eigenvector would exaggerate the variance measure.

For $λ_{2} = 0.5$ :

[1 - 0.5 0.5 0.5 1 - 0.5] [x y] = [0.5 0.5 0.5 0.5] [x y] = [00]

From the first row: $0.5 x + 0.5 y = 0$ → $x = - y$ .

The direction is $(1, - 1)$ . Normalize:

w^{(2)} = [\frac{1}{2} - \frac{1}{2}]

Dimensionality check: Both eigenvectors are in $R^{2}$ (matching $D = 2$ ) and have unit norm. The two eigenvectors are perpendicular — we verify this formally in Step 7.

17.4.6 Step 5: Projecting the Data

Rationale: The final step transforms the data onto the new axes. Each data point gets new coordinates — its projection lengths along each principal direction.

The transformed data is $Z = XW$ , where $W$ contains both eigenvectors as columns:

W = [\frac{1}{2} \frac{1}{2} \frac{1}{2} - \frac{1}{2}]

Z = - 1 01 - 1 10 [\frac{1}{2} \frac{1}{2} \frac{1}{2} - \frac{1}{2}]

Multiply it out row by row (the result is $3 \times 2$ ):

Row 1: $(- 1) (1/ 2) + (- 1) (1/ 2) = - 2/ 2 = - 2$ , and $(- 1) (1/ 2) + (- 1) (- 1/ 2) = 0$

Row 2: $(0) (1/ 2) + (1) (1/ 2) = 1/ 2$ , and $(0) (1/ 2) + (1) (- 1/ 2) = - 1/ 2$

Row 3: $(1) (1/ 2) + (0) (1/ 2) = 1/ 2$ , and $(1) (1/ 2) + (0) (- 1/ 2) = 1/ 2$

So the transformed data:

Z = - 2 1/ 2 1/ 2 0 - 1/ 2 1/ 2

The first column $Z_{1}$ is the projection onto the first principal component; its direction is $(1/ 2, 1/ 2)$ . The second column $Z_{2}$ is the projection onto the second; its direction is $(1/ 2, - 1/ 2)$ .

Shape check: $X$ is $3 \times 2$ , $W$ is $2 \times 2$ , so $Z$ is $3 \times 2$ . Correct — three data points, two transformed features.

17.4.7 Step 6: Deciding Whether to Reduce Dimensions

Rationale: Not every PCA justifies dimension reduction. If the eigenvalues are similar, all directions carry comparable variance — dropping any component loses significant information.

Compute the variance proportions:

Total variance = λ_{1} + λ_{2} = 1.5 + 0.5 = 2.0 = 100%

Proportion from Z_{1} = \frac{1.5}{2.0} = 0.75 = 75%

Proportion from Z_{2} = \frac{0.5}{2.0} = 0.25 = 25%

The first component captures 75% of the variance. The second captures 25%. Reducing from 2 dimensions to 1 would lose 25% of the information, which is usually unacceptable. Dimension reduction is not worthwhile here.

If one eigenvalue had been very small (say 0.02), capturing only 1% of variance, you could confidently drop it. You would work with just $Z_{1}$ .

17.4.8 Step 7: Orthogonality — Why New Features Are Uncorrelated

Check the dot product of the two eigenvectors:

w^{(1)} \cdot w^{(2)} = (\frac{1}{2}) (\frac{1}{2}) + (\frac{1}{2}) (- \frac{1}{2}) = \frac{1}{2} - \frac{1}{2} = 0

The eigenvectors are orthogonal. Since the new axes are perpendicular, any data projected onto them yields uncorrelated scores. $Z_{1}$ and $Z_{2}$ have zero linear correlation.

This is the most important property of PCA: it transforms correlated inputs into uncorrelated outputs. PCA removes collinearity by construction.

Q: If we have 4 or 5 features, will all the new $Z$ vectors still be perpendicular to each other?

A: Yes. Every pair of eigenvectors of a symmetric matrix (like the covariance matrix) is orthogonal. All $Z_{1}, Z_{2}, \dots, Z_{D}$ are mutually perpendicular. That is a fundamental property of PCA — the output features are always uncorrelated. This follows from the Spectral Theorem: a real symmetric matrix has an orthonormal basis of eigenvectors.

17.4.9 Complexity, Cost, and When to Use Alternatives

Computing the full PCA has two main costs:

Covariance matrix: Computing $X^{⊤} X$ takes $O (N D^{2})$ operations — you multiply an $N \times D$ matrix by its $D \times N$ transpose.
Eigendecomposition: Finding all eigenvalues and eigenvectors of a $D \times D$ symmetric matrix costs $O (D^{3})$ using standard algorithms.

For large $D$ (e.g., 20,000 features), the $O (D^{3})$ cost is prohibitive. This is the motivation for the Gram matrix trick (Section 17.6), which swaps the problem to an $N \times N$ matrix when $N ≪ D$ . In practice, libraries like numpy (via np.linalg.eigh) and scikit-learn use optimized LAPACK routines that are much faster than the theoretical $O (D^{3})$ bound, but the cubic scaling in $D$ still dominates for high-dimensional data.

Use the full eigendecomposition approach when:

$D$ is moderate (say, $D < 5000$ ) and you need all principal components.
You are doing PCA for the first time on a dataset and want to inspect the full eigenvalue spectrum to choose $k$ .

Use alternatives when:

$D$ is very large and $N ≪ D$ : use the Gram matrix trick (Section 17.6) or SVD.
You only need the top $k$ eigenvectors, not all $D$ : use iterative methods (power iteration, Lanczos) which cost $O (k \cdot nnz (X))$ — much cheaper.
You are using scikit-learn: PCA(n_components=k, svd_solver='arpack') automatically uses an iterative solver for the top $k$ components.

Pitfalls in the PCA computation.

Forgetting to mean-center. Computing $X^{⊤} X$ or $C$ without subtracting the mean gives the second-moment matrix, not the covariance matrix. The eigenvectors will be wrong.
Using $N$ instead of $N - 1$ . The course standard is $N - 1$ (sample covariance). Using $N$ gives different eigenvalues and can cost marks.
Not normalizing eigenvectors. PCA requires unit eigenvectors. If you skip normalization, the projection lengths $Z = X W$ will be scaled incorrectly, and the "variance = eigenvalue" equality breaks.
Computing $Z$ without checking orthogonality. Always verify that $w^{(1)} \cdot w^{(2)} = 0$ as a sanity check. A nonzero dot product means you made an arithmetic error in the eigenvector computation.
Assuming PCA always reduces dimensions. If the eigenvalues are all similar, dimension reduction loses real information. Always compute the proportion of variance before deciding to drop components.

Recap: The full PCA computation has seven steps:

Center the data.
Build the covariance matrix.
Find eigenvalues.
Find eigenvectors and normalize them.
Project the data: $Z = X W$ .
Compute variance proportions.
Verify orthogonality.

This is the hand-computation workflow you need for exams. Bridge: But we have been viewing PCA only through the lens of maximizing variance. There is a second, equivalent perspective: minimizing reconstruction error. The next section shows why these two goals are mathematically identical — and how the geometry of projection ties them together.

17.5 The Error Minimization Perspective

So far, PCA has been about one thing: find the direction where the data spreads out the most. But why should "maximum spread" produce good compressed features? Because there is a hidden second goal that maximum spread automatically achieves: minimum reconstruction error. When you project data onto a line and then try to rebuild the original points from those projections, the PCA direction gives you the smallest possible rebuilding mistakes. These two goals — maximize variance and minimize reconstruction error — are mathematically identical. Every PCA eigenvector is a two-for-one deal.

Think of taking a photograph of a 3D object — say, a coffee mug. You can only capture a 2D image. If you shoot from the front, you see the handle sharply but lose the depth. If you shoot from the side, you lose the logo. A good 2D projection looks as close as possible to the real 3D object — minimizing the reconstruction error (the information lost by flattening). PCA does the same thing: it finds the camera angle where a flat projection looks most like the real high-dimensional data. The reason this angle also maximizes "spread" is that the more of the object you capture, the more variation is visible in the photo. The two goals align perfectly.

Symbol registry for this section:

Symbol	Meaning	LaTeX	Type
$x_{i}$	Original data point	$x_{i} \in R^{D}$	vector
$x^_{i}$	Projected data point onto direction $w$	$x^_{i} = (x_{i}^{⊤} w) w$	vector
$w$	Unit direction vector	$w, ∥ w ∥ = 1$	vector
$∥ \cdot ∥$	Euclidean norm (length)	$∥ v ∥ = \sum v_{j}^{2}$	scalar

17.5.1 Projection Geometry

PCA has a second, equivalent interpretation: minimizing reconstruction error. Instead of asking "which direction has the most spread?", you ask "which direction lets me rebuild the original data most accurately after projection?"

Take a data point $x_{i}$ . When you project it onto a direction $w$ (a unit vector), the projected point — its "shadow" on that line — is:

x^_{i} = (x_{i}^{⊤} w) w

Let's unpack this. $x_{i}^{⊤} w$ is the dot product — a scalar. It measures how far along $w$ the original point lies. It is the coordinate of the projection. Multiplying this scalar by $w$ places the point back in the original $D$ -dimensional space, but now it sits exactly on the line through $w$ . The vector $x^_{i}$ is the orthogonal projection — the point on the line that is closest to $x_{i}$ .

Walkthrough with numbers. Let $x_{i} = (2, 1)$ and the raw direction be $(- 1, 1)$ . To project correctly we must use a unit vector, since the orthogonality property only holds when $∥ w ∥ = 1$ .

The vector $(- 1, 1)$ has length $2$ , so the normalized direction is $w = (- 1/ 2, 1/ 2)$ .

Dot product (the projection coordinate):

x_{i}^{⊤} w = (2) (- 1/ 2) + (1) (1/ 2) = - 1/ 2

Now place it back on the line:

x^_{i} = (- \frac{1}{2}) (- \frac{1}{2}, \frac{1}{2}) = (\frac{1}{2}, - \frac{1}{2})

Check that the displacement is perpendicular to the line:

x_{i} - x^_{i} = (2 - \frac{1}{2}, 1 - (- \frac{1}{2})) = (\frac{3}{2}, \frac{3}{2})

(\frac{3}{2}) (- \frac{1}{2}) + (\frac{3}{2}) (\frac{1}{2}) = 0 ✓ Perpendicular.

The geometric insight is what matters. The projection lands at the foot of a perpendicular dropped from the point to the line — the closest point on the line to $x_{i}$ .

Sense-check: The projected point $(1/2, - 1/2)$ should be closer to $(2, 1)$ than any other point on the line. Pick another point on that line, say $(2, - 2)$ : its distance to $(2, 1)$ is $(2 - 2)^{2} + (- 2 - 1)^{2} = 3$ . The distance from $(2, 1)$ to the projection $(1/2, - 1/2)$ is $(3/2)^{2} + (3/2)^{2} = 4.5 \approx 2.12$ . The projection is indeed closer. $✓$

17.5.2 Maximizing Variance Equals Minimizing Error

The projection error — the reconstruction error — is the distance between the original point and its projection:

Error_{i} = ∥ x_{i} - x^_{i} ∥

By the Pythagorean theorem, since $x^_{i}$ is the orthogonal projection (the displacement is perpendicular to the line):

∥ x_{i} ∥^{2} = ∥ x^_{i} ∥^{2} + ∥ x_{i} - x^_{i} ∥^{2}

The total squared length of the original point equals the squared length of its projection plus the squared error. This is a right triangle: the original vector is the hypotenuse, the projection is one leg, and the error vector is the other leg.

Now sum over all data points:

i = 1 \sum N ∥ x_{i} ∥^{2} = i = 1 \sum N ∥ x^_{i} ∥^{2} + i = 1 \sum N ∥ x_{i} - x^_{i} ∥^{2}

The left-hand side, $\sum ∥ x_{i} ∥^{2}$ , is the total sum of squares of the data — a constant. It does not depend on your choice of projection direction $w$ . The decomposition therefore holds for any $w$ .

(constant) = (variance of projections) + (reconstruction error)

This is the key insight. To make the reconstruction error as small as possible, you must make the variance of projections as large as possible. Maximizing variance and minimizing reconstruction error are the exact same optimization problem — just stated from two angles, literally two sides of the same right triangle.

The companion derivation (from the standard mathematical treatment) makes this even more explicit. The reconstruction error for a single point expands as:

∥ x_{i} - (x_{i}^{⊤} w) w ∥^{2} = (x_{i} - (x_{i}^{⊤} w) w)^{⊤} (x_{i} - (x_{i}^{⊤} w) w) = x_{i}^{⊤} x_{i} - 2 (x_{i}^{⊤} w)^{2} + (x_{i}^{⊤} w)^{2} (w^{⊤} w) = x_{i}^{⊤} x_{i} - (x_{i}^{⊤} w)^{2}

Since $w^{⊤} w = 1$ , the expression simplifies. Summing over all points:

Total Error = i = 1 \sum N ∥ x_{i} ∥^{2} - i = 1 \sum N (x_{i}^{⊤} w)^{2}

The second term, $\sum (x_{i}^{⊤} w)^{2}$ , is precisely the sum of squared projection lengths — proportional to the variance of the projected data (since data is centered). Minimizing total error is equivalent to maximizing this variance term. Both perspectives lead to the same eigenvalue problem $C w = λ w$ .

Visual intuition. Imagine a 2D scatterplot — a cloud of points. Draw a line through the origin and rotate it. For each angle:

Projections: Drop a perpendicular from each point to the line. Where it lands is the projected point $x^_{i}$ . The spread of these landing points along the line is the variance.
Errors: Measure the perpendicular distance from each original point to the line. Sum these squared distances. That is the reconstruction error.

As you rotate toward the PCA direction (the first eigenvector), two things happen simultaneously: the landing points spread further apart (variance increases), and the perpendicular distances shrink (error decreases). At the optimal angle — aligned with the dominant eigenvector — variance is maximal and error is minimal.

Rotate past the optimal angle: the landing points cluster closer (variance drops), and the perpendicular distances grow (error rises). The two quantities trade off perfectly because they sum to a constant.

Picture the right triangles: Each data point, its projection, and the error vector form a right triangle. The data point's length (hypotenuse) is fixed. When you rotate the line, the projection leg grows or shrinks. Since the hypotenuse is fixed, when the projection leg grows, the error leg must shrink. PCA finds the direction that makes all the projection legs as long as possible — which makes all the error legs as short as possible, simultaneously for all points.

Assumptions and scope.

Orthogonal projection: The equivalence relies on the projection being orthogonal (perpendicular to the line). PCA's solution is specifically the orthogonal projection. If you used a non-orthogonal projection, the Pythagorean decomposition would not hold and the equivalence would break.
Centered data: The constant total sum of squares $\sum ∥ x_{i} ∥^{2}$ is only the "total variance" if the data is mean-centered. Otherwise, the sum of squares includes a contribution from the mean, and the variance-error trade-off becomes muddied.
Euclidean distance: The error uses Euclidean (L2) distance. This is what makes the Pythagorean theorem applicable. If you measured error differently (L1, or any other norm), the equivalence would not hold — PCA specifically minimizes squared Euclidean reconstruction error.
Single direction at a time: The derivation shows the equivalence for one direction. For $k > 1$ directions, the same logic extends: the $M$ -dimensional principal subspace minimizes reconstruction error among all $M$ -dimensional linear subspaces.

Pitfalls.

Confusing the projected coordinate with the projected vector. The scalar $z_{i} = x_{i}^{⊤} w$ is the coordinate along $w$ . The vector $x^_{i} = z_{i} w$ is the actual point in the original space. PCA returns coordinates $z_{i}$ (stored in $Z$ ), not the projected vectors $x^_{i}$ .
Thinking PCA "throws away" the error directions. The discarded components (small eigenvalues) correspond to the orthogonal complement — the directions where reconstruction error would be large. PCA does not throw them away randomly; it keeps the components that contribute most to accurate reconstruction.
Forgetting that error minimization is for the centered data. Reconstruction error in the original (uncentered) space requires adding back the mean: $x ~_{i} = x^_{i} + x ˉ$ .
Assuming the Pythagorean decomposition always holds. It holds because PCA uses orthogonal projection onto a linear subspace. For nonlinear dimensionality reduction (t-SNE, autoencoders), there is no such clean decomposition — you cannot separate variance and error with a simple sum of squares.

Recap: Maximizing variance and minimizing reconstruction error are equivalent because of the Pythagorean theorem: total squared length = projection squared length + error squared length, and total length is fixed. PCA finds the direction where both goals are optimally satisfied — the two perspectives are one mathematical truth. Bridge: Both the variance-max and error-min perspectives require computing eigenvectors of the $D \times D$ covariance matrix. But what if $D$ is enormous — tens of thousands of features — and $N$ is small? The Gram matrix trick in the next section shows how to solve PCA when $N ≪ D$ by working with an $N \times N$ matrix instead.

17.6 The Gram Matrix Trick: When N Is Much Smaller Than D

Standard PCA crashes when you have more features than data points. A 20,000 × 20,000 covariance matrix takes gigabytes of memory and hours to eigendecompose — if it fits in RAM at all. But here is the surprising trick: the eigenvectors you want also live, hidden, inside a much smaller matrix. If you only have 50 data points, you can solve PCA on a 50 × 50 matrix instead and then mathematically recover the full 20,000-dimensional eigenvectors. This is the Gram matrix trick, and it is the computational backbone of PCA in genomics, text analysis, and any domain where features vastly outnumber samples.

Imagine you need to find the tallest person in a room. You could measure everyone directly — that is $O (D^{3})$ PCA. Or you could ask each person to stand next to a reference pole and note the comparison. If there are only a few people (small $N$ ) but the "measuring" is complicated (large $D$ ), the comparison approach is faster. The Gram matrix trick reverses the problem: instead of wrestling with a $D \times D$ covariance matrix, you compare data points to each other and recover the PCA directions from those comparisons. The analogy breaks at the recovery step — the reconstructed directions are exact, not approximate. This is a mathematical identity, not a heuristic.

Symbol registry for this section:

Symbol	Meaning	LaTeX	Type
$N$	Number of data points (rows)	$N$	integer
$D$	Number of features (columns)	$D$	integer
$K$	Gram matrix	$K = X X^{⊤}$	$N \times N$ matrix
$v$	Eigenvector of $K$	$v \in R^{N}$	vector
$w$	PCA direction (eigenvector of $X^{⊤} X$ )	$w \in R^{D}$	vector
$λ$	Eigenvalue (shared by $K$ and $X^{⊤} X$ )	$λ$	scalar

17.6.1 Wide Form vs. Long Form Data

Form	Condition	Example	Shape
Long form	$N ≫ D$	10,000 samples, 50 features	Tall matrix
Wide form	$N ≪ D$	50 samples, 20,000 features	Wide matrix

Wide form data is common in genomics (thousands of gene expressions, few patients), text analysis (thousands of word frequencies, few documents), and sensor fusion (many sensor readings, few time points). The standard PCA covariance matrix $X^{⊤} X$ would be $D \times D$ . At $20, 000 \times 20, 000$ , storing it as 64-bit floats occupies about $20, 00 0^{2} \times 8 bytes = 3.2 GB$ . Computing its eigenvectors costs $O (D^{3})$ — roughly $8 \times 1 0^{12}$ operations — which is impractical on commodity hardware.

Real-world: In genomics, a typical microarray study has $N = 100$ patients and $D = 20, 000$ gene expressions. The covariance matrix would be $20, 000 \times 20, 000$ . The Gram matrix approach swaps this for a $100 \times 100$ eigendecomposition — a speedup factor of about $(200)^{3} / (1)^{3} \approx 8, 000, 000 \times$ in the eigendecomposition alone.

17.6.2 Solving $X X^{⊤}$ Instead of $X^{⊤} X$

Start from the eigenvalue equation for PCA (ignoring the $1/ (N - 1)$ factor — it does not affect the eigenvectors):

X^{⊤} Xw = λ w

Here $w$ is a $D$ -dimensional eigenvector of the $D \times D$ matrix $X^{⊤} X$ . Pre-multiply both sides by $X$ :

X X^{⊤} (Xw) = λ (Xw)

Define $v = Xw$ (an $N$ -dimensional vector) and $K = X X^{⊤}$ (an $N \times N$ matrix). Then:

Kv = λ v

This is again an eigenvalue equation, but now for the $N \times N$ Gram matrix $K$ instead of the $D \times D$ matrix $X^{⊤} X$ . The eigenvalue $λ$ is the same. The eigenvector $v$ is $N$ -dimensional instead of $D$ -dimensional.

For our wide-form example ( $N = 50, D = 20, 000$ ), $K$ is only $50 \times 50$ . Eigendecomposition of a $50 \times 50$ matrix takes microseconds. The $20, 000 \times 20, 000$ version would take hours — if it ran at all.

Notation note: The professor uses the data-as-rows convention ( $X \in R^{N \times D}$ ), so the Gram matrix is $K = X X^{⊤} \in R^{N \times N}$ . Some texts use data-as-columns ( $X \in R^{D \times N}$ ), which flips the dimensions. The key idea is the same: swap the order of multiplication to get the smaller matrix.

17.6.3 Recovering the PCA Directions

Once you have the eigenvectors $v$ and eigenvalues $λ$ of the Gram matrix $K = X X^{⊤}$ , recover the original PCA direction $w$ using:

w = \frac{X ^{⊤} v}{λ}

Why $λ$ in the denominator? Because $w$ must be a unit vector ( $∥ w ∥ = 1$ ). The proof:

∥ X^{⊤} v ∥^{2} = (X^{⊤} v)^{⊤} (X^{⊤} v) = v^{⊤} X X^{⊤} v = v^{⊤} Kv = v^{⊤} (λ v) (since v is an eigenvector of K) = λ ∥ v ∥^{2} = λ (since v is a unit vector)

So $∥ X^{⊤} v ∥ = λ$ . Dividing by $λ$ normalizes the result to unit length. This gives the PCA direction $w$ without ever touching the $D \times D$ covariance matrix.

Dimensionality check: $X^{⊤}$ is $D \times N$ , $v$ is $N \times 1$ , so $X^{⊤} v$ is $D \times 1$ — a vector in the original feature space. Correct.

Summary of the Gram trick — algorithmic recipe:

Compute the Gram matrix: $K = X X^{⊤}$ (size $N \times N$ ).
Eigendecompose $K$ to get eigenvectors $v^{(j)}$ and eigenvalues $λ_{j}$ .
Recover PCA directions: $w^{(j)} = X^{⊤} v^{(j)} / λ_{j}$ .
Project data if needed: $Z = X W$ as usual.

This is the foundation of the kernel trick (where inner products in a high-dimensional feature space are computed implicitly) and connects directly to SVD.

Connection to SVD. The Singular Value Decomposition factorizes $X = UΣ V^{⊤}$ (with data-as-rows convention). Then:

X^{⊤} X = V Σ^{2} V^{⊤}, X X^{⊤} = U Σ^{2} U^{⊤}

The columns of $V$ are eigenvectors of $X^{⊤} X$ — these are the PCA directions $w$ in data-as-rows convention. The columns of $U$ are eigenvectors of $X X^{⊤}$ — these are our $v$ vectors. The singular values $σ_{i}$ relate to eigenvalues via $λ_{i} = σ_{i}^{2}$ (up to the $1/ (N - 1)$ factor). So the Gram matrix trick is essentially performing PCA through the SVD lens, solving the smaller of the two eigenproblems. In scikit-learn, PCA(svd_solver='auto') internally uses SVD on the data matrix rather than eigendecomposing the covariance matrix, which is numerically more stable and avoids forming $X^{⊤} X$ explicitly.

Q: Is this similar to SVD where wide-form data is converted to a long-form problem?

A: Yes, it is the same principle. SVD decomposes $X = UΣ V^{⊤}$ . Then $X^{⊤} X = V Σ^{2} V^{⊤}$ and $X X^{⊤} = U Σ^{2} U^{⊤}$ . The columns of $V$ are the eigenvectors of $X^{⊤} X$ — these are our $w$ vectors (PCA directions). The columns of $U$ are the eigenvectors of $X X^{⊤}$ — these are our $v$ vectors. The Gram matrix trick is doing PCA through the SVD lens — solving the smaller eigenproblem and recovering the larger one mathematically. The nonzero eigenvalues of $X^{⊤} X$ and $X X^{⊤}$ are identical.

Scope and practical limits.

The Gram matrix $K$ must be positive semi-definite. With $N < D$ and no duplicate points, $K$ has rank $N$ (full rank) and is invertible — all eigenvalues are positive. If data points are duplicated, $K$ becomes rank-deficient and some eigenvalues are zero.
Recovery assumes $λ > 0$ . The formula $w = X^{⊤} v / λ$ requires the eigenvalue to be nonzero. Zero eigenvalues correspond to directions with zero variance — they are irrelevant for PCA anyway.
Memory for $X$ still scales with $D$ . Even though you eigendecompose an $N \times N$ matrix, you still need to store $X$ ( $N \times D$ ) and compute $X^{⊤} v$ ( $D \times N$ times $N \times 1$ ). For extreme $D$ (e.g., millions), even storing $X$ can be a bottleneck. In those cases, kernel PCA or randomized SVD methods are preferred.
The trick works for any $N, D$ . There is no requirement that $N < D$ — if $N > D$ , standard PCA is more efficient. The trick is specifically motivated by the $N ≪ D$ case.

Pitfalls.

Forgetting the normalization. If you skip $λ$ , the recovered $w$ will not be a unit vector. The "variance = eigenvalue" relationship breaks, and the projection $Z = X W$ will have incorrectly scaled values.
Confusing $X^{⊤} X$ and $X X^{⊤}$ . $X^{⊤} X$ (size $D \times D$ ) contains feature-feature covariances. $X X^{⊤}$ (size $N \times N$ ) contains data point-data point similarities. If you eigendecompose the wrong one, your eigenvectors will have the wrong dimensionality.
Not mean-centering before the Gram trick. The relationship $Kv = λ v$ with $K = X X^{⊤}$ and $v = Xw$ holds for the mean-centered $X$ . If you do not center, the recovered $w$ are not PCA directions — they are directions of maximum second moment (pointing toward the data centroid).
Using the Gram trick when $N > D$ . If $N = 10, 000$ and $D = 50$ , the Gram matrix is $10, 000 \times 10, 000$ — much larger than the $50 \times 50$ covariance. The Gram trick helps only when $N < D$ .

Recap: When $N ≪ D$ , swap the eigenproblem: eigendecompose the $N \times N$ Gram matrix $K = X X^{⊤}$ instead of the $D \times D$ covariance $X^{⊤} X$ . The eigenvalues are identical; recover PCA directions via $w = X^{⊤} v / λ$ . This is the computational backbone of PCA in high-dimensional-low-sample settings and connects directly to SVD and the kernel trick. Bridge: The remaining section consolidates industry wisdom — when and why PCA matters in practice, beyond the mathematics.

17.7 Cautionary Notes and Industry Wisdom

You now understand PCA mathematically and computationally. But the gap between knowing PCA and using PCA effectively is where real projects succeed or fail. This section is about judgment — when to reach for PCA, when to put it back on the shelf, and why understanding your data matters more than mastering any single algorithm.

17.7.1 PCA Is Linear Feature Engineering

PCA is a feature engineering technique. The new features $Z_{j}$ are linear combinations of the originals:

Z_{j} = a_{j 1} X_{1} + a_{j 2} X_{2} + \dots + a_{j D} X_{D}

This is not a weakness — feature engineering is often where the biggest accuracy gains come from. In many competitions and real projects, cleverly constructed features improve performance more than switching from one model to another. But you must understand what kind of features you are creating: they are linear, uncorrelated, and ranked by variance.

Linear models (linear regression, logistic regression, linear SVM) benefit from PCA-transformed features because they work in the same linear space — PCA just rotates and decorrelates it. Nonlinear models (neural networks, random forests, kernel SVMs) may not benefit, or may be actively harmed. Neural networks already create nonlinear feature combinations in their hidden layers — they do implicit feature engineering through compositions of activation functions. Feeding them PCA-transformed data strips away the nonlinear structure their hidden layers were designed to discover.

17.7.2 Automating ML Pipelines Does Not Remove the Need for Understanding

Modern ML pipelines are increasingly automated — AutoML agents select algorithms, preprocess data, and tune hyperparameters. But these agents follow heuristics, not understanding. An agent might blindly apply PCA before a neural network because "PCA reduces dimensions" is in its training data. It does not know that PCA destroys the nonlinear signal the network needs.

Someone must supervise these agents and verify their decisions. That someone is you. Knowing when to apply which technique — and why — is what separates effective practitioners from those who only run .fit() and .predict(). The key message: automation replaces execution, not judgment. The pipeline does not think; you do.

17.7.3 Spend Time on Data, Not Just Models

The most impactful work in a machine learning project is understanding your data. Know its structure, correlations, nonlinearities, and what features actually drive the target variable. PCA is one exploratory tool among many — a lens for seeing linear structure. Feature engineering (of which PCA is one form) deserves more attention than model selection. A well-understood dataset with thoughtfully engineered features will outperform a poorly understood dataset fed into the most sophisticated model.

A practical hierarchy to follow: data understanding > feature engineering > model selection > hyperparameter tuning. Spend your time accordingly.

That hierarchy sets up the single most important habit to carry out of this lecture.

Recap — Lecture capstone. PCA is linear. It transforms correlated features into uncorrelated ones ranked by variance. It works before linear models, it fails on nonlinear data, and it should never be applied blindly. The Gram trick makes it computationally feasible when $N ≪ D$ . The two perspectives — maximum variance and minimum reconstruction error — are mathematically identical. Know the full computation by hand for exams. Know the practical rules for real projects. And always, always center your data first.

Exam Guidance Summary

Problem types to expect:

1. Eigenvalue-based feature reduction: Given eigenvalues $λ_{1}, λ_{2}, \dots, λ_{D}$ , find how many components capture a given percentage of variance. Compute total variance $\sum λ_{i}$ . Then compute each proportion $λ_{k} / \sum λ_{i}$ . Compute cumulative sums. Find the smallest $k$ that exceeds the threshold (typically 90% or 95%).

Exam note: This problem type tests whether you understand that eigenvalues equal variance captured and that cumulative proportion determines component count. Common mistake: forgetting to compute total variance before dividing.

2. Full PCA computation: Given a small data matrix (typically $2 \times 2$ or $3 \times 3$ after centering, or a small $N \times D$ with $N = 3$ or $4$ ), complete the pipeline:

Mean center the data (subtract column means).
Compute the covariance matrix $C = \frac{1}{N - 1} X^{⊤} X$ .
Find eigenvalues via $det (C - λ I) = 0$ .
Find normalized eigenvectors via $(C - λ I) w = 0$ .
Project the data: $Z = X W$ .
Compute variance proportions and decide if dimension reduction is worthwhile.

Exam note: This is the most heavily weighted problem type. Practice the full computation on at least three different datasets until you can reproduce every step without referring to notes. Show all intermediate steps — partial credit depends on visible work.

3. Conceptual questions:

When does PCA work? (linear relationships) When does it fail? (nonlinear: circles, spirals, curved manifolds)
What is the relationship between eigenvalues and variance? ( $λ_{i}$ = variance captured by component $i$ )
Why are transformed features uncorrelated? (eigenvectors of a symmetric matrix are orthogonal)
What is the alternative when $N ≪ D$ ? (Gram matrix trick: eigendecompose $X X^{⊤}$ instead of $X^{⊤} X$ )
How are maximizing variance and minimizing reconstruction error related? (Pythagorean theorem — they sum to a constant)

Key points that lose marks:

Using $N$ instead of $N - 1$ for the covariance denominator. The course standard is $N - 1$ (sample covariance). Two conventions exist — $N$ for population, $N - 1$ for sample. Use $N - 1$ .
Forgetting to mean-center the data before computing covariance. Eigenvectors without centering point toward the origin, not along directions of maximum spread.
Not normalizing eigenvectors to unit length. The constraint $∥ w ∥ = 1$ is fundamental — unnormalized eigenvectors break the "variance = eigenvalue" relationship.
Not showing all intermediate computation steps. Even if your final answer is correct, missing steps lose partial credit. Show every multiplication, every sum.

Study advice:

Go through the worked PCA computation (Section 17.4) until you can reproduce every step from scratch — mean centering, covariance, eigenvalues, eigenvectors, projection, variance proportions, and orthogonality check.
Practice eigenvalue and eigenvector calculations by hand for $2 \times 2$ and $3 \times 3$ matrices. Be comfortable with the characteristic equation $det (C - λ I) = 0$ .
Understand the conceptual chain: covariance = linear association → eigenvalue = variance captured → eigenvector orthogonality = uncorrelated features → maximizing variance = minimizing projection error.
Know the Gram matrix shortcut for wide-form data ( $N ≪ D$ ) and its connection to SVD.

Key Industry Applications

Self-driving cars: Cameras, Lidar, radar, temperature sensors, and wind sensors generate thousands of features that must be processed in real time. PCA identifies the high-information features for obstacle detection, lane tracking, and path planning, enabling perception pipelines to run faster with negligible loss of critical signal.
Oil extraction (Halliburton/Schlumberger): Geological surveys produce 273+ features per well site — chemical composition, rock porosity, seismic readings, pressure gradients. PCA compresses these to ~75 features capturing 97% of the variance. The resulting model improved R² from 58% to 60%. At millions of dollars per well, a 2% yield prediction improvement represents massive operational savings. Even a 0.5% accuracy gain justifies the effort.
Content platforms (Netflix/YouTube): User behavior generates high-dimensional feature vectors — watch history, search queries, device type, time of day, demographic signals. PCA identifies which features carry the most predictive signal for viewing duration, enabling better ad placement and content recommendations without processing every raw feature.
Standard pipeline for high-dimensional data: PCA first removes linear redundancy (1000 → 70 features). Then t-SNE or encoder-decoder extracts nonlinear structure (70 → 10 features). The final model trains on the 10-feature compressed representation. This staged approach respects both the linear and nonlinear structure in the data.
Genomics and bioinformatics: Gene expression datasets routinely have $D > 20, 000$ genes and $N < 500$ patients. The Gram matrix trick makes PCA feasible by swapping the huge $20, 000 \times 20, 000$ covariance for a small $500 \times 500$ similarity matrix. PCA-compressed gene signatures help with cancer subtyping, drug response, and biomarker discovery.
Agent-based ML automation: As AutoML tools become widespread, they may misapply PCA — for example, applying it before a neural network because "reduce dimensions" is a generic rule in their training data. Human supervision requires understanding why PCA + neural network is the wrong combination, and when to override the automated pipeline's decisions.

MFML Lecture 17 notes · Principal Component Analysis — Practical Computation and the Dual Perspective

Mathematical Foundations for Machine Learning· Undergraduate· 2026-07-11

Sections Breakdown

1PCA — Practical Applications, Worked Computation, and the Dual Perspective

217.1 PCA Recap and Motivation

317.2 When PCA Works and When It Does Not

417.3 The Practical PCA Pipeline

517.4 Worked PCA Computation

617.5 The Error Minimization Perspective

717.6 The Gram Matrix Trick: When N Is Much Smaller Than D

817.7 Cautionary Notes and Industry Wisdom

9Exam Guidance Summary

10Key Industry Applications

University students studying the mathematical foundations of machine learning

Exam Revision Notes

Below is the distilled, exam-ready core of this lecture. Every entry is built from the full textbook notes above. Use this section for rapid review — but if something doesn't make sense, go back to the full explanation in the main content.

PCA Recap: Variance Maximization via Eigenvectors

Must-know: PCA finds the unit directions that maximize projected variance. The solution is the eigenvectors of the covariance matrix; the eigenvalue is exactly the variance that direction captures.

$C w = λ w, w^{o} pC w = λ$

⚠️ Top pitfall: Forgetting to mean-center first. Without centering, the eigenvectors point toward the origin, not along the directions of maximum spread.

Self-check: Why is the eigenvalue equal to the captured variance, not just proportional to it?

Connects to: Worked PCA computation, variance-equals-reconstruction-error, the Gram matrix trick.

When PCA Works and When It Fails

Must-know: PCA is built on covariance, a linear measure. It works on linear/cigar-shaped data and fails on circles, spirals, and curved manifolds where covariance is near zero despite strong relationships.

$Cov (X_{j}, X_{k}) = \frac{1}{N - 1} i = 1 \sum N x_{ij} x_{ik}$

⚠️ Top pitfall: Assuming zero covariance means independence. It only means no linear association — a quadratic relationship can have covariance exactly zero.

Self-check: For data on a circle, what do the eigenvalues look like, and can you reduce dimensions safely?

Connects to: Nonlinear alternatives (t-SNE, autoencoders), the practical pipeline.

PCA Removes Multicollinearity

Must-know: The new features are mutually uncorrelated by construction because eigenvectors of a symmetric matrix are orthogonal. This makes $X^{o} pX$ invertible again for linear regression.

$w^{(i)} \cdot w^{(j)} = 0 (i e q j), Z = X W, Z^{o} pZ e x t d ia g o na l$

⚠️ Top pitfall: Dropping correlated features by hand from a correlation heatmap — correlation only sees linear dependence and you may destroy nonlinear predictive signal.

Self-check: Why does decorrelating inputs fix a non-invertible $X^{o} pX$ ?

Connects to: PCA + linear regression, ridge regression, the variance threshold.

PCA Before Linear vs Nonlinear Models

Must-know: PCA is a linear transform. It helps linear models (regression, logistic, linear SVM). It harms neural nets and adds little for trees, because it strips the nonlinear structure those models exploit.

$Z_{j} = a_{j 1} X_{1} + a_{j 2} X_{2} + \dots + a_{j D} X_{D}$

⚠️ Top pitfall: Applying PCA blindly before every model. Know your downstream model: skip PCA for neural nets, random forests, and RBF-kernel SVMs.

Self-check: A neural net already learns nonlinear combinations in its hidden layers — what does PCA destroy for it?

Connects to: Multicollinearity, the standard industry pipeline.

Choosing the Variance Threshold

Must-know: Keep the smallest $k$ such that the cumulative proportion of variance exceeds your threshold (typically 90-95%). Because eigenvalues are sorted descending, the top- $k$ is always optimal.

$Proportion_{k} = \frac{\sum _{i = 1}^{k} λ _{i}}{\sum _{i = 1}^{D} λ _{i}}$

⚠️ Top pitfall: Forgetting to divide by the total variance (sum of all eigenvalues) before computing proportions — a classic mark-loser.

Self-check: Given eigenvalues 3, 2, 1, 0.5, how many components capture 90% of variance?

Connects to: Worked computation, deciding whether to reduce dimensions.

Full Worked PCA Computation

Must-know: The exam hand-computation is a fixed seven-step recipe: center, build $C = r a c 1 N - 1 X^{o} pX$ , find eigenvalues, find and normalize eigenvectors, project $Z = X W$ , compute variance proportions, verify orthogonality.

$C = \frac{1}{N - 1} X^{o} pX, det (C - λ I) = 0, Z = X W$

⚠️ Top pitfall: Using $N$ instead of $N - 1$ , or skipping eigenvector normalization. Both break the "variance = eigenvalue" identity and cost marks.

Self-check: After finding eigenvectors, what dot product must equal zero as a sanity check?

Connects to: PCA recap, variance threshold, orthogonality.

Maximizing Variance Equals Minimizing Error

Must-know: By the Pythagorean theorem, total squared length = projection squared + error squared. Total length is fixed, so maximizing projection variance is identical to minimizing reconstruction error.

$∥ x_{i} ∥^{2} = ∥ x^_{i} ∥^{2} + ∥ x_{i} - x^_{i} ∥^{2}, x^_{i} = (x_{i}^{o} p w) w$

⚠️ Top pitfall: Confusing the scalar coordinate $z_{i} = x_{i}^{o} p w$ with the projected vector $x^_{i} = z_{i} w$ . PCA stores coordinates, not the vectors.

Self-check: Why does the equivalence require orthogonal projection and centered data?

Connects to: Projection geometry, the Gram matrix trick.

The Gram Matrix Trick (N Much Smaller Than D)

Must-know: When features vastly outnumber samples ( $N ≪ D$ ), eigendecompose the small $N im es N$ Gram matrix $K = X X^{o} p$ instead of the huge $D im esD$ covariance, then recover directions via $w = X^{o} p v / λ$ .

$K v = λ v, w = \frac{X ^{o} p v}{λ}$

⚠️ Top pitfall: Forgetting the $λ$ normalization, or using the trick when $N > D$ (where it makes things worse). Also remember to center first.

Self-check: Why are the nonzero eigenvalues of $X X^{o} p$ and $X^{o} pX$ identical?

Connects to: SVD, the kernel trick, complexity of PCA.

PCA Is Linear Feature Engineering

Must-know: PCA is a feature-engineering step. It produces linear, uncorrelated, variance-ranked features. It helps when you understand your data. Automation replaces execution, not judgment about when to use it.

$Z_{j} = a_{j 1} X_{1} + a_{j 2} X_{2} + \dots + a_{j D} X_{D}$

⚠️ Top pitfall: Letting AutoML blindly apply PCA before a neural net because "dimension reduction" is a generic rule — it destroys the nonlinear signal the net needs.

Self-check: Where does the professor rank data understanding versus model selection in project priority?

Connects to: Linear vs nonlinear models, the industry pipeline.

Previous LectureLecture 16 · Principal Component Analysis

Next LectureLecture 18 · Quadratic Programming and the KKT Conditions

Was this lecture useful?

Loading comments…

🤖

BitsNotes AI Assistant

Subject Notes Assistant

Configure AI Chat

Choose how to access the chatbot

Have your own API key?

Switch to "Bring Your Own Key" tab above for unlimited access with any OpenAI-compatible provider.

AI Provider Preset

API Endpoint URL (OpenAI-Compatible)

API Key

Model Name

🔑 Enter API key above to fetch live models from provider, or enter model name manually.

Remember key for this session (until tab closes)

OpenAI-Compatible API Support

Choose any provider preset (Gemini, DeepSeek, Kimi, GLM, MiniMax, Qwen, OpenAI, Groq, Ollama, etc.) or enter a custom endpoint URL.

Security & Privacy First

Your API key is sent directly from your browser to your specified provider. BitsNotes servers never store or see your key.

Prerequisite Knowledge

Previously Covered in This Subject

17.1 PCA Recap and Motivation

17.1.1 Recap: Maximizing Variance via Eigenvectors

17.1.2 Symbol Registry — PCA Eigenvalue Formulation

17.1.3 Why Dimension Reduction Matters

17.2 When PCA Works and When It Does Not

17.2.1 Covariance Measures Linear Association

17.2.2 PCA Fails on Nonlinear Data

17.2.3 Beyond PCA: t-SNE and Encoder-Decoder

17.3 The Practical PCA Pipeline

17.3.1 PCA Removes Multicollinearity

17.3.2 PCA + Linear Regression Works; PCA + Neural Networks Does Not

17.3.3 The Standard Industry Pipeline — A Procedural Workflow

17.3.4 The Pitfall of Manual Correlation-Based Removal

17.3.5 Choosing the Variance Threshold

17.4 Worked PCA Computation

17.4.1 Step 1: Mean Centering

17.4.2 Step 2: Covariance Matrix

17.4.3 Symbol Registry — Worked PCA Computation

17.4.4 Step 3: Finding Eigenvalues

17.4.5 Step 4: Finding Eigenvectors

17.4.6 Step 5: Projecting the Data

17.4.7 Step 6: Deciding Whether to Reduce Dimensions

17.4.8 Step 7: Orthogonality — Why New Features Are Uncorrelated

17.4.9 Complexity, Cost, and When to Use Alternatives

17.5 The Error Minimization Perspective

17.5.1 Projection Geometry

17.5.2 Maximizing Variance Equals Minimizing Error

17.6 The Gram Matrix Trick: When N Is Much Smaller Than D

17.6.1 Wide Form vs. Long Form Data

17.6.2 Solving XX⊤ Instead of X⊤X

17.6.3 Recovering the PCA Directions

17.7 Cautionary Notes and Industry Wisdom

17.7.1 PCA Is Linear Feature Engineering

17.7.2 Automating ML Pipelines Does Not Remove the Need for Understanding

17.7.3 Spend Time on Data, Not Just Models

Exam Guidance Summary

Key Industry Applications

MFML Lecture 17 notes · Principal Component Analysis — Practical Computation and the Dual Perspective

Sections Breakdown

Exam Revision Notes

PCA Recap: Variance Maximization via Eigenvectors

When PCA Works and When It Fails

PCA Removes Multicollinearity

PCA Before Linear vs Nonlinear Models

Choosing the Variance Threshold

Full Worked PCA Computation

Maximizing Variance Equals Minimizing Error

The Gram Matrix Trick (N Much Smaller Than D)

PCA Is Linear Feature Engineering

Was this lecture useful?

BitsNotes AI Assistant

Configure AI Chat

Sign in to use AI Chat

17.6.2 Solving $X X^{⊤}$ Instead of $X^{⊤} X$