Generative Modeling Goals and Principal Component Analysis

These notes cover two opening themes of unsupervised deep learning. First: what it means to learn from raw data without labels, including the self-supervision trick that lets data carry its own training signal. Second: why squeezing many columns of data into a few informative ones matters, and how principal component analysis (PCA) does it step by step — all the way to eigenfaces, where PCA runs on images of faces.

The lecture builds in four moves. Sections 2.1–2.2 set the mission: learning without labels and the goal of generative modeling. Sections 2.3–2.4 tour the classical dimensionality-reduction toolbox and motivate why reduction helps at all. Section 2.5 develops PCA end to end — mean, covariance, eigenvectors, projection, choosing the component count, reconstruction, and the power method. Section 2.6 puts PCA to work on face images. Classroom questions and corrections appear where they belong in the flow, because each one marks a spot where understanding can slip.

2.1 Deep Networks Learning Without Labels

2.1.1 The Label-Free Principle

The core promise is easy to state. We take raw data and process it with deep networks — stacked function approximators such as multi-layer perceptrons (MLPs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), and transformers. We do this in a label-free way: the training set supplies inputs only, with no human-assigned target labels.

In symbols, the training set is just $\{{\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_N\}$, a list of $N$ data items where each ${\mathbf{x}}_i$ is one feature vector. A supervised course pairs every ${\mathbf{x}}_i$ with a target $y_i$; here no $y_i$ ever shows up. That missing column is the whole identity of this subject.

There is one twist, and it carries the lecture. Sometimes we use the training data in a smart way so the data acts as a stand-in for labels. The data — or a transformation of the data — plays the role the labels would normally play. This style of training is called self-supervised learning, and it still counts as label-free because no extra annotation is ever collected. If you took a prior deep neural network course, you already used this idea there.

A concrete everyday picture helps. Think of a student alone with a textbook and no answer key. They cover half a sentence with a hand, guess the hidden words from the visible context, then slide the hand away to check. The book graded its own quiz. Self-supervision runs exactly this trick at scale: the input hides part of itself, predicts it back, and scores itself against the true hidden part. Where does the analogy break? A student tires after twenty questions; the training loop never does. It plays millions of such quizzes per hour and stores every improvement in the network weights.

2.1.2 Self-Supervision in Recurrent Models

The cleanest demonstration uses an RNN trained to predict the next word or next character. Take the sentence "sun is shining". Feed the words as input; the desired output at each step is the next word, ending with an end-of-sentence marker after "shining". You create every target by shifting the input one position forward. The input generates its own supervision.

A fill-in-the-gap task works the same way. Given "Yesterday ___ sun shining", the model reads the words before and after the gap and predicts what belongs inside. The raw sentence provides everything; no annotator is needed. Both tasks share one design rule: the target must exist inside the input itself.

Real-world: this shifting trick powers everyday text prediction on phone keyboards, and it is the pretraining objective behind modern language models. The next-token game, played over trillions of words, is what teaches those systems grammar, facts, and style before any customer ever types a prompt.

2.1.3 When Labels Are Still Necessary

Not every recurrent-network task escapes labels, and sorting this out was the first quiz of the day.

So the dividing line reads: when the target lives inside the input — next word, missing word — self-supervision wins. When the target is an external judgment — sentiment class, grammatical role, translated sentence — someone must supply labels, because no amount of shifting manufactures them.

The pattern generalizes far past this quiz. Generative modeling, clustering, and dimensionality reduction all live on the label-free side; sentiment tags and star ratings live on the other side. Keeping the two sides separate is the first habit of this course.

Real-world and domain connection: deciding whether labels are needed is the first scoping question in any applied machine-learning project. A team building a support-ticket router must budget for labeled tickets; a team building autocomplete does not. Same architecture family, opposite data economics — the dividing line above decides the budget line.

2.2 Generative Modeling

2.2.1 Matching a Model Distribution to the Data Distribution

The heart of this course is generative modeling. Picture all your data — images, text, audio — sitting inside one big cloud. Images dominate such pictures because a glance reveals what an image contains, while thirty sentences of text demand real reading time; text also makes the modeling harder.

We want a model written $p_{\text{model}}$: a formula, an analytical and statistical function whose independent variables are the features of the data. Those features might be individual pixel values, individual words, or word embeddings of a text. The goal is to tune $p_{\text{model}}$ so this model density sits very close to the true data density $p_{\text{data}}$:

$$p_{\text{model}}(\mathbf{x})\approx p_{\text{data}}(\mathbf{x})\text{for data items }\mathbf{x}$$

Here $\mathbf{x}$ is one data item written as a feature vector, $p_{\text{data}}$ is the unknown real-world distribution that produced the dataset, and $p_{\text{model}}$ is our tunable stand-in for it. Many techniques in this course are just different routes to building $p_{\text{model}}$ accurately. Once trained, you sample from it and create new data similar to — but not identical to — the originals.

An everyday analogy for the whole enterprise: think of a chef who spends a year eating one cuisine — say, Thai cooking. Nobody hands the chef recipes. After a year of tasting, the chef can cook new dishes nobody has written down, yet every dish still tastes unmistakably Thai. Training plays the role of the year of tasting; sampling plays the role of cooking a dish no recipe describes. Where the analogy breaks: the chef brings intention and creativity, while the model only reproduces the statistics it absorbed — ask it for Ethiopian food and it keeps drifting back toward what it tasted.

Two restrictions shape everything that follows:

1. Sampling must be tractable: drawing a new sample from $p_{\text{model}}$ should be computationally easy.
2. You can plug any sample back into $p_{\text{model}}$ and read off its probability.

The second restriction gives probability meaning. A high value says "usual, plausible sample"; a low value says "rare novelty". Move far from the center of the distribution and you generate very novel data — so novel that users may fail to relate it to the originals. Keep that trade-off in mind whenever a generative system produces strange outputs.

Why bother? Learn a great $p_{\text{model}}$ and you can compress: keep the model, discard the giant dataset, and regenerate data on demand. You can also build generative AI applications of the kind GPT-family systems offer. At the core of those products sits exactly this ability to model probability distributions. Prompt engineering decides the initial input; the learned distribution shapes what gets generated. This is not an applied prompt-engineering course, but it supplies the foundation beneath one.

2.2.2 Worked Illustration: Sampling From One Gaussian

How does sampling from a learned distribution actually run? Take a one-dimensional Gaussian, purely for ease of explanation. Learning the distribution means estimating its mean $\mu$ — say the empirical mean is zero — and its spread $\sigma$.

For reference, the Gaussian density itself reads:

$$p(x)=\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp \left(-\frac{(x-\mu {)}^2}{2{\sigma }^2}\right)$$

where $x$ is the value being scored, $\mu$ centers the bell, $\sigma$ sets its width, and the leading fraction guarantees the curve integrates to exactly 1 — the property that makes it a valid probability density.

Any probability density integrates to 1, which is why a uniform draw between 0 and 1 fits the next step perfectly. Build the cumulative distribution function (CDF) $F$ of the Gaussian: it rises from 0 and flattens toward 1, climbing fastest at the peak of the density. Now draw a random number $u$ from a uniform distribution over $[0,1)$ and feed it through the inverse of that curve. This inverse cumulative transform turns uniform draws into Gaussian samples:

$$u\sim \mathrm{Uniform}(0,1),x=F^{-1}(u)$$

where $F^{-1}$ denotes the inverse CDF — you hand it a probability mass and it returns the point below which that much probability lies. For a general Gaussian with mean $\mu$ and standard deviation $\sigma$, sample the standard normal first and then set $x=\mu +\sigma F_{0,1}^{-1}(u)$, where $F_{0,1}$ is the standard normal CDF.

Most uniform draws land where the CDF climbs steeply — exactly the crowded middle of the bell. A single fitted Gaussian may still model real data poorly; you might need a mixture of Gaussians or something far more flexible. Whatever the family, the demands stay fixed: learn the distribution as accurately as possible, train fast, sample fast, and keep the samples diverse rather than repetitive. Diversity is a defining criterion of a good generative system.

No single technique wins on all counts today. Some models are more accurate; some train faster; some sample faster. So we study a broad set of models instead of betting on one favorite — breadth beats betting on a single champion, and the leader changes every couple of years anyway. Imperfect density models are also where nonsense generations and hallucinations creep in.

Often you want not just a sample but also its likelihood — how probable that sample is, whether rare or routine. Different techniques expose different subsets of these abilities, so always ask which abilities a given model actually provides.

2.2.3 Continuous Versus Discrete Data

Most statistics training lives on continuous distributions: exponential, Gaussian, mixtures of Gaussians. Images cooperate with that worldview — pixel intensities behave like continuous quantities, so density estimation feels natural.

Text is discrete, but assumptions bridge the gap through quantization — representing values with a finite alphabet. Later in the course, around the midterm, we meet the variational autoencoder (VAE) and its vector-quantized variant, which is very effective for generating new text. We will also see diffusion models that run on a discrete latent space, again useful for text. The scope stretches past images: text from images, images from text, and molecules too.

That exchange marks the honest boundary of unsupervised generation: a model of faces makes faces, not tax forms.

2.2.4 Molecules, Medicine, and Honest Limits

Generating molecules is a serious industrial application. Drug candidates are molecules; creating one consumes enormous time and money with no revenue guarantee. A few blockbuster drugs subsidize the many research programs that never land. Think of established blood-pressure and cholesterol medicines, aspirin-era classics, and newer protein-based treatments such as Ozempic. Proposing candidate molecules with generative models is a big deal in pharmaceuticals. Molecules map naturally onto graphs with nodes and links; the graph neural network course covers generating graph-structured data.

Real-world: pharmaceutical firms treat molecule generation as essential infrastructure, not a productivity toy. Text tools, by contrast, mostly serve productivity — summaries and drafts rather than finished writing — and that productivity game carries labor-economics consequences familiar to anyone in services.

Domain connection and recap: generative modeling turns a pile of unlabeled examples into a samplable probability distribution — the same move behind drug-candidate proposals at pharmaceutical firms, image synthesis products, and the language models that power modern chat systems. With the goal fixed, the next sections step back to the classical toolbox: before deep networks entered, dimensionality reduction already solved half of this game by compressing data while keeping its structure.

2.3 A Map of Classical Dimensionality Reduction

2.3.1 Principal Component Analysis and Its Practical Cousins

Before deep networks enter, this stretch of the course tours classical techniques — no neural networks here. Their shared goal: build features, using mathematical or statistical moves, that capture data records in a compact form. Principal component analysis (PCA) anchors the tour; an earlier machine-learning course introduced it together with covariance matrices, eigenvalues, and eigenvectors, so treat this as a re-encounter, not a first meeting. The full step-by-step construction arrives in Section 2.5.

Three engineered relatives matter once scale bites:

1. Randomized PCA — uses randomness tricks to approximate the leading components when an exact decomposition grows too slow. Full singular value decomposition on an $N\times D$ matrix costs on the order of $O(N{D}^2)$ work plus $O(D^3)$, while randomized schemes target only the top few components and finish far earlier. You trade a little accuracy for a lot of speed.
2. Incremental PCA — updates components as data arrives; built for streaming sources or memory-bound machines. Instead of loading the whole dataset, you feed small batches and let each batch nudge the existing components.
3. Sparse PCA — pushes components toward few nonzero weights for interpretability. A component that loads on three features reads like a sentence; one that loads on two thousand reads like static.

All three exist so that very large datasets — data at rest or data in motion — still enjoy principal components. If a machine emits fresh readings every five minutes, incremental PCA absorbs each batch without recomputing from scratch.

2.3.2 Kernel PCA for Non-Linear Structure

Plain PCA hunts axes of maximal variation, then second-maximal, third, and onward. It shines when data forms one elongated, single-clump cloud: the scatter shows a clear longest axis. Trouble begins with multimodal data — several separated clumps. The axes PCA picks then miss the cluster structure entirely, because variance is a poor guide when the interesting question is which clump a point belongs to.

The word implicit deserves unpacking, because it is where the magic hides. You never compute coordinates in the huge new space. Every formula PCA needs — variances, projections — can be rewritten in terms of dot products between data points, and the kernel trick replaces each dot product ${\mathbf{x}}_i^{\top }{\mathbf{x}}_j$ with a kernel function $k({\mathbf{x}}_i,{\mathbf{x}}_j)$ that behaves like a dot product in the lifted space. So you get the geometry of the high-dimensional world while paying only the cost of evaluating $k$ in the original one.

When to pick which: start with vanilla PCA; reach for kernel PCA when the scatter shows curves, rings, or interleaved clumps that a straight line cannot describe.

So kernel PCA delivers dimensional reduction plus retained separability, motivated exactly like the kernel SVM.

2.3.3 Independent Component Analysis: Unmixing Sources

Independent component analysis (ICA) suits data that is not unimodal. Its headline application is blind source separation. Imagine one microphone in a room holding a radio, a television, and talking people. The microphone records one signal — a superposition of everything playing at once. ICA breaks that single mixture into its pieces: the radio stream, the TV stream, the voices. Nothing about the sources is known in advance — that is what makes it blind.

Why is separating even possible? Because independent signals mix additively, and a sum of independent signals tends to look more bell-shaped (more Gaussian) than any individual source — the same averaging effect behind the classic result that sums of many small random influences look Gaussian. ICA turns that fact around: it searches for directions along which the unmixed signal looks as non-Gaussian and as statistically independent as possible, because those directions must be the original sources peeking through.

This also draws the exam-relevant line against PCA. PCA asks for directions of maximal variance and returns uncorrelated outputs; ICA asks for statistically independent sources and returns the signals themselves. Two sound sources with equal power are a job for ICA — maximal-variance directions would just blend them again.

Real-world connection: the same mathematics cleans up audio in hearing aids and conference-call systems, and separates overlapping brain-signal channels in EEG headsets used by neuroscientists.

2.3.4 Canonical Correlation Analysis: Coupling Two Views

Canonical correlation analysis (CCA) is a linear method linking two views of the same event. Take someone speaking on camera. One view is the audio track; the other is the video track — specifically how pixels change around the lip region. CCA finds a vector space for the audio and a vector space for the video such that projecting the signals onto these spaces maximizes the correlation between them. Audio and video hold no mystery here — both are just vectors of numbers, so projection into lower-rank vector spaces makes sense.

In symbols: given paired views $\mathbf{a}\in {\mathbb{R}}^p$ (audio features) and $\mathbf{v}\in {\mathbb{R}}^q$ (video features), CCA chooses projection directions ${\mathbf{w}}_a$ and ${\mathbf{w}}_v$ to maximize the correlation between the projected pair:

$$\underset{{\mathbf{w}}_a,{\mathbf{w}}_v}{\max }\mathrm{corr}\left({\mathbf{w}}_a^{\top }\mathbf{a},{\mathbf{w}}_v^{\top }\mathbf{v}\right)$$

where corr denotes the Pearson correlation between the two projected sequences over the training pairs, ${\mathbf{w}}_a\in {\mathbb{R}}^p$, and ${\mathbf{w}}_v\in {\mathbb{R}}^q$. Several such direction pairs can be found, each capturing one channel of shared information between the views.

Applications follow. Train on paired speech and lip movement, then generate lip movement from audio alone — one format mapped onto another, in the same spirit as translating English into German. Push further for dubbing: take English audio, convert it to Hindi or Tamil, then synthesize lip movement matching the new language — no actors needed for reshoots. CCA also opens the door to multimodal information fusion, treated fully in a later-semester course.

Nearby companions in this toolbox include locally linear embeddings, and for text there is Latent Semantic Indexing (LSI), which derives topic-linked features from bag-of-words counts — a document's word counts go in, topic-flavored numbers come out.

Domain connection: dubbing studios and accessibility-tool vendors use exactly this audio-to-lip coupling to localize video content across languages, and search engines historically used LSI-style reduction to match queries to documents by topic rather than exact words.

2.4 Why Reduce Dimensions At All?

2.4.1 Making Data Visible and Thinkable

Dimensionality reduction swaps the 200 original columns for perhaps the 5 most informative ones. Those new columns are usually linear combinations of old ones, not copies of any single column, yet patterns reappear and visualization becomes possible again. The payoff is insight: you can finally see how the high-dimensional data spreads, clusters, and drifts.

You can plot freely up to three dimensions — a scatter on paper, or a rotatable 3D view. Beyond three, plot selected pairs or triples of the reduced variables. Cumbersome but workable, and usually enough to gain insight into how the full-dimensional cloud behaves.

2.4.2 Worked Example: Engineering the Body Mass Index

Reduction is not always eigen-machinery; sometimes one clever ratio suffices.

A note on definitions, since reference charts differ: this lecture treats the index as the simple ratio weight over height with an obesity signal above 25 and a healthy band near 15–20 that moves with age. Most adult health references instead define BMI as weight in kilograms divided by the square of height in meters — for the patients above, $95/1.70^2\approx 32.9$ versus $50/1.60^2\approx 19.5$ — with standard bands of underweight below 18.5, normal up to about 25, and obesity at 30 and beyond. Keep whichever convention your assessment uses, but know both exist.

Better representation leads to better visualization, better reasoning, and better downstream classification or pattern recognition — the same record set became easier to understand purely because someone chose smarter features.

2.4.3 Noise, Weak Directions, and Hidden Factors

Raw data often carries noise: uncorrelated perturbations that randomly nudge every observation in every direction. Structure concentrates in strong directions — those are the ones where measurements genuinely co-vary. Weak-energy directions tend to hold noise, because noise spreads thinly everywhere instead of piling up along any axis. Reduce onto the strong dimensions and much of the noise falls away with the discarded weak ones.

Reduction can also reveal latent variables — hidden causes, absent from the collected columns, that drive what you observe. Psychometric questionnaires produce hundreds of parameter readings; behind them stand interpretable factors such as ego, personality, and intelligence. Those factors guide the visible values even though nobody measured them directly. Likewise, a document's bag-of-words vector feeds Latent Semantic Indexing, which yields numbers tied to the topics the text discusses — the topics were never columns in the table, yet they explain the word counts.

This reframes what reduction is for. You are not just shrinking tables; you are hunting for the small set of hidden dials that generate the large set of visible readings.

2.4.4 The Reconstruction Test

What separates honest reduction from careless column-dropping? The reconstruction test.

Once trusted, the small representation feeds classification, clustering, and pattern recognition directly — and the smaller feature set means far less training data is needed, because a learner facing 50 features needs far fewer examples than one facing 500.

Domain connection: clinical labs run exactly this play when they compress dozens of correlated blood markers into a handful of risk indices, and streaming services compress a viewer's watch history into a few taste coordinates before recommending the next show.

2.5 Principal Component Analysis, Step by Step

2.5.1 Notation and Setup

Let $\mathbf{x}\in {\mathbb{R}}^D$ be one data point: every entry is a real number, and there are $D$ of them. Two concrete anchors keep the notation honest. A $20\times 10$ grayscale image flattens into a 200-number vector, so $D=200$. A text record's bag-of-words vector has one slot per vocabulary word — easily tens of thousands of slots.

Stack $N$ such points row-wise into the data matrix $X\in {\mathbb{R}}^{N\times D}$: row $i$ holds point ${\mathbf{x}}_i$, so the matrix has one row per record and one column per feature. PCA seeks a mapping from $D$ dimensions down to $k$ dimensions — written with a small $k$ to stress $k\ll D$ — that survives the reconstruction test of Section 2.4.4: compress, rebuild, compare against the original, and accept only a close match.

2.5.2 The Mean Vector and the Covariance Matrix

Everything starts with two summaries. The sample mean averages every record feature-wise:

$$\mathbf{\mu }=\frac{1}{N}\sum _{i=1}^N{\mathbf{x}}_i$$

where $\mathbf{\mu }\in {\mathbb{R}}^D$ collects one average per feature: entry $d$ of $\mathbf{\mu }$ is the average of column $d$ over all records. Subtracting $\mathbf{\mu }$ from every row moves the data cloud so its center sits at the origin — PCA assumes centered data, and centering is how we honor that.

The covariance matrix measures how features co-vary. The recipe: subtract the mean from every row, multiply each centered row by its own transpose (an outer product), and average over the record count:

$$\Sigma =\frac{1}{N}\sum _{i=1}^N({\mathbf{x}}_i-\mathbf{\mu })({\mathbf{x}}_i-\mathbf{\mu }{)}^{\top }$$

where $\Sigma$ is the covariance matrix and $(\cdot {)}^{\top }$ marks the transpose. Check the shapes: each ${\mathbf{x}}_i-\mathbf{\mu }$ is $D\times 1$, its transpose is $1\times D$, so every outer product is $D\times D$, and their average stays $D\times D$. Entry $(r,c)$ of $\Sigma$ records how feature $r$ moves with feature $c$; the diagonal entry $(d,d)$ is just the variance of feature $d$.

So variance is the one-dimensional special case — how far points spread around the mean along a single axis — while covariance generalizes it to how two or more dimensions vary together, captured as a matrix.

A classroom quiz pinned the shapes down for a concrete dataset.

Physical data adds a signature: covariance matrices usually look banded. Bands mean neighboring features correlate strongly — adjacent sensor readings, adjacent pixels, adjacent time samples tend to move together because they come from one coupled physical process. Data generated by physics carries its coupling straight into $\Sigma$.

2.5.3 Reading the Shape of the Cloud

Covariance signs tell direction stories. When $x$ rises and $y$ rises together across records, covariance is positive. When $x$ rises but $y$ sinks, covariance is negative. Zero means no linear co-movement.

Two structural facts make $\Sigma$ special. First, it is symmetric: swapping $r$ and $c$ changes nothing, since "$x_r$ moving with $x_c$" is the same statement as "$x_c$ moving with $x_r$" — formally, entry $(r,c)$ equals entry $(c,r)$ because the outer product $({\mathbf{x}}_i-\mathbf{\mu })({\mathbf{x}}_i-\mathbf{\mu }{)}^{\top }$ is already symmetric. Second, it is positive semi-definite, meaning every direction $v$ satisfies $v^{\top }\Sigma v\ge 0$. Read that expression as "the spread of the cloud along direction $v$" — an average of squared projections, which can never be negative.

Together these two facts force the eigenvectors to come out real and mutually orthogonal. Here is the algebra behind the orthogonality claim, for two eigenvectors with distinct eigenvalues:

$$\begin{array}{rlrl}{\lambda }_i{\mathbf{u}}_i^{\top }{\mathbf{u}}_j& =(\Sigma {\mathbf{u}}_i{)}^{\top }{\mathbf{u}}_j& & \text{(substituting }\Sigma {\mathbf{u}}_i={\lambda }_i{\mathbf{u}}_i\text{)}\\ & ={\mathbf{u}}_i^{\top }{\Sigma }^{\top }{\mathbf{u}}_j& & \text{(transpose of a product)}\\ & ={\mathbf{u}}_i^{\top }\Sigma {\mathbf{u}}_j& & \text{(symmetry: }{\Sigma }^{\top }=\Sigma \text{)}\\ & ={\lambda }_j{\mathbf{u}}_i^{\top }{\mathbf{u}}_j& & \text{(substituting }\Sigma {\mathbf{u}}_j={\lambda }_j{\mathbf{u}}_j\text{)}\end{array}$$

So $({\lambda }_i-{\lambda }_j){\mathbf{u}}_i^{\top }{\mathbf{u}}_j=0$. If the eigenvalues differ, the dot product must be zero — the directions stand perpendicular. Symmetry also guarantees the eigenvalues themselves are real, and positive semi-definiteness guarantees they are non-negative, exactly what quantities named "variance" require.

Now the geometry. An elliptically scattered cloud has a major axis and a minor axis. The ellipse follows the classic conic form

$$\frac{x^2}{a^2}+\frac{y^2}{b^2}=1$$

where $a$ and $b$ are the semi-axis lengths: setting $y=0$ gives the $x$-intercepts $\pm a$, and setting $x=0$ gives the $y$-intercepts $\pm b$. (Start from a unit circle $x^2+y^2=1$ and stretch it by a factor $a$ horizontally and $b$ vertically — the stretched curve is exactly this equation.) The quantities $a$ and $b$ tie directly to the covariance matrix of elliptically distributed data: big spread along one direction shows up as a large variance there. Pictured examples made it concrete: one cloud with variances around 0.75 against 0.25 traces a long, tapered ellipse; a perfectly round cloud owns a purely diagonal covariance — identical diagonal entries, zeros elsewhere — and no direction is privileged.

Correlation matrices normalize covariance and stay symmetric: divide each covariance by the product of the two features' standard deviations, and the diagonal becomes ones while off-diagonal entries become fractions between $-1$ and $1$. Those fractions encode the same geometry as the covariance numbers, just on a fixed scale.

2.5.4 Eigenvalues, Eigenvectors, and the SVD Route

The objects of interest satisfy the eigen equation:

$$\Sigma {\mathbf{u}}_j={\lambda }_j{\mathbf{u}}_j$$

where ${\mathbf{u}}_j\in {\mathbb{R}}^D$ is the $j$-th eigenvector — a direction in feature space that $\Sigma$ only stretches — and ${\lambda }_j\ge 0$ is its eigenvalue, a scalar reporting how much variance lives along that direction. Read the equation as a question: along which directions does the covariance act like a pure stretch? Those directions organize the cloud. For elliptical data, the largest eigenvalue ${\lambda }_1$ belongs to the major axis and ${\lambda }_2$ to the minor axis, provided the cloud really is elliptically distributed. Sketching the top two eigenvectors over the scatter reproduces the two axes of maximal variation.

Compute these directions with the singular value decomposition (SVD) or with the power method of Section 2.5.8. The decomposition writes:

$$\Sigma =US{V}^{\top }$$

where $U$ stacks eigenvector directions as columns, $S$ is a diagonal matrix carrying the singular values — the eigenvalue magnitudes ${\lambda }_1$ through ${\lambda }_D$ along its diagonal, sorted largest first — and $V^{\top }$ is the transposed right factor. Because the covariance matrix is symmetric, its left and right singular directions coincide: $V=U$, so the decomposition collapses to the eigendecomposition $\Sigma =US{U}^{\top }$. That identity is why several orderings — $US{V}^{\top }$, $U^{\top }SV$, and friends — all appeared interchangeable in lecture: for a symmetric matrix they describe the same factorization read forward or backward, and any of them hands you the same eigenvectors.

For physically generated data the eigenvalue list decays fast: the first few are large, the rest tiny. That decay is the whole reason reduction works, and it decides which eigenvectors survive as principal components.

2.5.5 Projection: Two Dimensions Become One Number

Take a two-dimensional cloud and target one dimension. First compute the leading eigenvector ${\mathbf{e}}_1$. Then project every point:

$$z_1={\mathbf{e}}_1^{\top }(\mathbf{x}-\mathbf{\mu })$$

where $\mathbf{x}\in {\mathbb{R}}^D$ is the original point, $\mathbf{\mu }$ the mean, and $z_1$ the single new coordinate — the dot product drops the centered point onto the axis and returns one scalar lying between negative and positive extremes. Dot product is projection: for two arrows from the origin, it measures how far one reaches along the other. Every $D$-dimensional point becomes one number.

Continuing the tiny dataset above: projecting the point $\mathbf{x}=(2,1)$ gives $$z_1=\frac{1}{\sqrt{2}}(2)+\frac{1}{\sqrt{2}}(1)=\frac{3}{\sqrt{2}}\approx 2.12,$$ and its mirror point $(-1,-2)$ lands at $z_1=(-1-2)/\sqrt{2}\approx -2.12$. Two coordinates in, one number out — and the pair still orders the points sensibly along the cloud's long direction.

With $k$ kept components the rules tighten: each ${\mathbf{u}}_i$ has unit length, and distinct components stay perpendicular, ${\mathbf{u}}_i^{\top }{\mathbf{u}}_j=0$ for $i\ne j$ — together an orthonormal set. Stacking the chosen eigenvectors as rows of a matrix $A\in {\mathbb{R}}^{k\times D}$, the whole operation compresses to:

$$\mathbf{y}=A(\mathbf{x}-\mathbf{\mu })$$

where $\mathbf{y}\in {\mathbb{R}}^k$ is the new reduced point, $\mathbf{x}\in {\mathbb{R}}^D$ the old one, and the rows of $A$ are precisely the kept eigenvectors. Shape check: $(k\times D)(D\times 1)=k\times 1$, as promised.

What the small codes buy: plots when $k\le 3$, pairwise panels when larger; machine-learning algorithms that run far faster on $k$ features than on $D$; smaller training-set needs, because fewer features require fewer examples; and storage savings — $k$ numbers per record instead of $D$.

2.5.6 Choosing k: The Explained Variance Ratio

How many components are enough? Reduction must not discard important information, and importance is measured by eigenvalues. Keep components with large eigenvalues; drop those tied to tiny ones. Formally, the explained variance ratio weighs the kept eigenvalues against the total energy:

$$\frac{\sum _{j=1}^k{\lambda }_j}{\sum _{j=1}^D{\lambda }_j}\ge 0.90$$

The numerator sums the kept eigenvalues; the denominator sums all of them — the total variance energy of the dataset. Pick the smallest $k$ clearing your threshold: 90%, 95%, or 99%. Which bar applies depends on the application: if a business task performs well at 90%, ride with the smaller $k$; demanding 99% pushes $k$ upward.

A quick picture of the decision: plot the running sum of explained variance against the number of components kept. The curve shoots up steeply, then bends into a flat tail — the bend, called the elbow, marks where extra components stop paying rent. The elbow position is a decent estimate of the data's true intrinsic dimensionality.

On the tiny worked dataset the rule fires immediately: ${\lambda }_1/({\lambda }_1+{\lambda }_2)=4.5/5.0=0.90$, so keeping one component clears the 90% bar exactly.

If all eigenvalues were large and none small, no effective reduction would exist. Physical data rescues us: strong correlations force fast decay, so a handful of components clears the bar. The class connection stuttered once, so the rule was stated a second time — and that closing restatement survives inside the exchange below.

2.5.7 Reconstructing the Original Data

With that settled, the inverse question follows naturally.

In equation form, reconstruction re-adds weighted eigen-directions on top of the mean:

$$\hat{\mathbf{x}}=\mathbf{\mu }+\sum _{j=1}^k{w}_j{\mathbf{u}}_j$$

where $w_j$ is the $j$-th projected score of the record (the number produced in Section 2.5.5) and ${\mathbf{u}}_j$ is its matching eigenvector. Each term slides one eigenvector-scaled brick back onto the foundation; $k$ bricks rebuild an approximation of the whole house. The board bookkeeping matches this exactly: the covariance factorizes so that a 200-by-200 object chains three thinner pieces — a 200-by-50 piece of eigenvector columns, a 50-by-50 diagonal of leading singular values, and a 50-by-200 transposed piece — whose product reconstructs the full-width object. Multiplying through, $(200\times 50)(50\times 50)(50\times 200)=200\times 200$: a rank-50 summary standing in for the full matrix.

The takeaway: reduced representation plus transposed basis equals original representation, record by record — and the gap between rebuilt and original is the reconstruction error of Section 2.4.4, made concrete.

2.5.8 The Power Method

One numerical engine deserves its name remembered: the power method — an iterative scheme that hunts the eigenvector belonging to the largest eigenvalue, then continues for the second-largest, third-largest, and onward, without ever computing a full decomposition.

The idea needs only repeated matrix-vector products. Start from almost any nonzero vector ${\mathbf{v}}_0$. Multiply by $\Sigma$ and renormalize, again and again:

$${\mathbf{v}}_{t+1}=\frac{\Sigma {\mathbf{v}}_t}{\Vert \Sigma {\mathbf{v}}_t\Vert }t=0,1,2,\dots$$

where ${\mathbf{v}}_t\in {\mathbb{R}}^D$ is the current guess, $\Sigma {\mathbf{v}}_t$ applies the covariance stretch, and the division by the norm $\Vert \cdot \Vert$ rescales the result back to unit length so nothing blows up. Why does this converge? Write ${\mathbf{v}}_0$ in the eigenbasis: it is a mix of all eigenvector directions. Each multiplication multiplies the component along ${\mathbf{u}}_j$ by ${\lambda }_j$, so after $t$ rounds the top-eigenvalue component outweighs its runner-up by a factor growing like $({\lambda }_1/{\lambda }_2{)}^t$ — which races to infinity unless the two leading eigenvalues are nearly tied. The surviving direction is ${\mathbf{u}}_1$. To find ${\mathbf{u}}_2$, remove the ${\mathbf{u}}_1$ contribution from the data (deflation) and repeat.

Cost check: each iteration costs one matrix-vector product, about $O(D^2)$ operations for a dense $D\times D$ matrix — far cheaper than a full decomposition when you only want the top few directions. The method slows to a crawl when ${\lambda }_1$ and ${\lambda }_2$ are nearly equal, because the decisive ratio approaches 1; that is also the case where PCA itself is ill-defined, since no single direction dominates.

Real-world connection: recommendation engines and text-retrieval systems face matrices too large for exact decompositions, and power-iteration-family algorithms — the engine inside Google's original PageRank among others — get the dominant directions cheaply.

2.6 Eigenfaces: PCA on Face Images

2.6.1 Computing and Reading Eigenfaces

The recipe runs exactly like Section 2.5, just on pictures. Take a dataset of $100\times 100$ grayscale portraits. Flatten each portrait into a 10,000-dimensional vector — one coordinate per pixel. Run PCA on the stacked matrix. Then draw the top eigenvectors back into their $100\times 100$ image shapes: those ghostly faces are the eigenfaces.

Reading them teaches geometry. First comes the average face — the mean vector wearing skin, the statistical center every other face bends around. Next, eigenfaces carrying significant eigenvalue magnitudes highlight what varies across people: overall shading, hairline position, prominent bone structure. Several look distinctly masculine in that dataset, because those were the strongest axes of variation it contained. One family of eigenfaces lights up on eyeglasses — wearers jump out along that axis. Descend to eigenvectors tied to small eigenvalues and the images break down into pure noise: nothing structured remains, because tiny-variance directions hold no consistent face information at all.

Pause on the caution these pictures provoke. Average-face templates skew toward the dataset's majority appearance, so classifiers built on them can lose accuracy for people with darker skin tones. That is how machine learning quietly perpetrates bias and prejudice — a central concern in practical deployments today, delivered here by an unintentional demonstration. The lesson generalizes far beyond faces: whenever a model's internal template is a population average, everyone far from the majority pays for it.

2.6.2 Recognition Pipeline and Compression Arithmetic

The recognition pipeline turns eigenfaces into identity decisions. It is a procedure worth memorizing step by step:

1. Flatten a new face into its 10,000-value vector.
2. Subtract the average face, centering it exactly like the training data was centered.
3. Project onto the significant eigenfaces — dot products, one per kept component.
4. Collect scores $w_1$ through $w_k$; these form the feature representation of the face.
5. Classify: an ordinary supervised classifier over those $k$ numbers decides which person the face belongs to.

A portrait that began as 10,000 pixel values now rides on roughly 200 numbers — and the classifier never needs the raw pixels again.

Rebuilding from a code sums weighted eigenfaces on top of the mean:

$$\hat{\mathbf{x}}=\mathbf{\mu }+w_1{\mathbf{u}}_1+w_2{\mathbf{u}}_2+\cdots +w_k{\mathbf{u}}_k\approx \text{90–95\% of the original face}$$

where $\mathbf{\mu }$ is the average face, each ${\mathbf{u}}_j$ an eigenface, each $w_j$ the face's score along it, and $\hat{\mathbf{x}}$ the rebuilt portrait. Quality scales brutally with how many terms you include.

The compression arithmetic makes the savings concrete. A $100\times 100$ grayscale face stores 10,000 bytes as raw pixels. Keeping 400 eigenface coefficients stores 400 bytes instead. Ten thousand divided by four hundred gives

$$\frac{10,000}{400}=25$$

— a 25-fold storage saving, with the face still recognizable. Compression, though, is only the bonus. The real prize: small feature spaces cut the training data a learner needs — datasets should run several times the feature count — so validation error improves accordingly. How far you can shrink while the downstream task still succeeds is the honest measure of a good reduction.

2.6.3 Patch-Based PCA: The Butterfly Experiment

Faces are small images; what about big scenes? Compression scales through patches.

Slice a large, high-resolution butterfly photograph — the lecture's image measured roughly 372 by 492 pixels, though only the patch size matters for the math — into $12\times 12$ patches. Each patch flattens to a vector of $12\times 12=144$ values, so the whole photograph becomes a pile of 144-dimensional vectors, one per patch location.

Run PCA on the pile and keep just 16 dimensions per patch.

The lesson generalizes well past butterflies: even aggressive low-dimensional codes preserve the gist that recognition needs, which is why patch-based coding ideas echo through modern vision pipelines.

Real-world connection: this is the same trick behind JPEG-style block compression and behind early face-recognition systems — both cut images into pieces, keep the dominant statistical patterns per piece, and discard the rest without losing what viewers (or classifiers) actually use.

Exam Guidance Summary

• A group assignment anchors the grading calendar. Teams have five members — a few have four or six, since the roster does not divide evenly. The task releases within two weeks, with about three weeks to finish. It covers PCA, its variants, and autoencoders, so complete group formation promptly once announced.
• The prescribed reading is the dimensionality-reduction chapter of Hands-On Machine Learning. Work through it diligently; it alone yields a complete picture, since much of this material follows that chapter's arc from PCA through its practical variants.
• Refresh CNN and RNN fundamentals from prior coursework — they recur as building blocks in the self-supervision discussion of Section 2.1 and later architectures.
• Ready-to-run notebooks accompany the posted materials. Execute them in Colab or a local setup before attempting problems; watching PCA run on real data cements the eigenvalue decay this lecture predicted on paper.
• Expect conceptual contrasts on assessments — vanilla versus kernel PCA, PCA versus ICA suitability, continuous versus discrete generation strategies — plus numeric drills styled like the dimension quiz (mean vector and covariance shapes for a 50,000-by-200 dataset) and the explained-variance computation (kept eigenvalue sum over total sum, clearing the threshold).

Key Industry Applications

• Pharmaceutical molecule generation — proposing drug candidates cuts discovery cost in a billion-dollar industry where a few blockbuster medicines subsidize many failed programs; molecule graphs fall to graph neural network methods, covered in a dedicated course.
• Dubbing and lip synthesis — CCA-trained audio-video coupling converts English audio into Hindi or Tamil speech with matched lip movement, no reshoots; localization studios and accessibility-tool vendors run on this coupling.
• Blind source separation — ICA unmixes radio, TV, and voices captured by one room microphone; the same idea cleans audio in hearing aids, conference-call systems, and EEG signal processing.
• Face recognition and compression — eigenfaces powered identity recognition while shrinking storage 25-fold in the worked example; the flatten-project-classify pipeline prefigures modern embedding-based recognition.
• Document intelligence — Latent Semantic Indexing turns bag-of-words vectors into topic-aware features for search and organization; queries match documents by theme rather than exact words.
• Generative AI products — GPT-style systems rest on learned distributions; prompt inputs steer sampling from them; text, image, and molecule generation all descend from this foundation.
• Gene expression analytics — PCA-style reduction tames thousands-of-genes measurements into analyzable structure, exposing the few latent factors that separate healthy from diseased samples.