PCA Variants and Autoencoders

3.1 Judging a Method Beyond Accuracy

First, a quick recap tying back to the previous session. When you reconstruct data with a large number of principal components, the result brings higher quality reconstruction; a small number of components loses fidelity, and the original comes back poorly recovered. That trade-off — more components buy quality but cost storage — is the backdrop for everything this session adds.

Before we study faster and smarter versions of PCA, we need to agree on how to judge any machine learning method. There are two main lenses, and every variant you meet below is best understood as a trade between them.

The first lens is task performance. For PCA this means reconstruction error: how well can the reduced data be turned back into the original data? For classifiers and regressors it means metrics you already know, such as the F1 score (a balance of precision and recall), validation error, and the various regression scores. A key habit from earlier courses is cross-validation: split the data into folds, train on some folds, test on the held-out part, then rotate which part is held out and average the results. Cross-validation confirms a result is statistically meaningful rather than a lucky split — it guards against numbers that look good by chance.

The second lens is computational load. This phrase hides several separate questions:

• Memory: does the dataset even fit in your machine's RAM? A method that needs the whole matrix in memory at once fails before it starts if the matrix does not fit.
• Time: how long does one full run take? Minutes are fine for exploration; hours change how your whole workflow is organized.
• Growth: how do memory and time scale when the dataset doubles? A method that is comfortable today may be impossible next quarter.

A method with great accuracy is useless if you cannot afford to run it. Most variants of PCA exist because of this second lens: plain PCA is expensive on big, high-dimensional data, and the practical fixes — randomized PCA, incremental PCA, and kernel PCA — each relax a different part of that cost.

3.1.1 Two Lenses: Performance and Computational Load

Keep both lenses in mind as a checklist. When someone proposes a new dimensionality reduction method, ask two questions: what happens to reconstruction error or downstream accuracy, and what happens to memory and time? Each PCA variant below trades a bit of one for a lot of the other.

Real-world: this two-lens checklist is how engineering teams pick solvers in practice. A recommendation system processing millions of user-item events nightly cares about time and memory first, because an exact method that misses its batch window is worthless regardless of accuracy; a medical imaging lab analyzing a few hundred scans may happily pay hours of compute for the most faithful reduction. Same mathematics, different binding constraint.

3.2 Computational Cost of Standard PCA

Start from the raw data. We collect it into a matrix with one row per training instance and one column per feature. Call the number of training points $m$ and the number of features $n$. So the data matrix has shape $m\times n$.

To run standard PCA on this matrix you must, in order:

1. Preprocess: remove the mean of each column and scale the columns so the data is properly normalized.
2. Form the covariance matrix: multiply the transposed data matrix with the data matrix itself.
3. Extract principal components: run singular value decomposition (SVD) on the result.

Each step touches the whole matrix, and the SVD step is the heavy one.

3.2.1 Where the Cost Comes From

There are two cost drivers. The first is forming the covariance matrix, which is a matrix multiplication whose work grows with both $m$ and $n$: every one of the $n^2$ output entries needs a sum over the $m$ rows. The second driver is the eigenvector extraction inside SVD, which grows with the cube of the feature count. On top of time, memory is a real limit: the covariance matrix alone holds $n\times n$ entries, so with hundreds of features and many thousands of rows the intermediate tables can be too big to hold in RAM at all.

An important subtlety: standard SVD computes every principal component. You usually do not know in advance how many components you need, because the right count only becomes visible once you see the full spectrum of eigenvalues. So you pay for all $n$ components even if you keep 200.

There is a cheaper habit for exploration: use an iterative eigensolver that finds one eigenvector at a time, largest eigenvalue first, then the second largest, and so on, stopping once reconstruction quality is acceptable. This solver is the power method: start from a random vector $v$, repeatedly multiply it by the matrix and renormalize, $v\leftarrow \frac{Av}{\Vert Av\Vert }$, and the vector swings toward the direction of the largest eigenvalue. Subtracting out ("deflating") each direction found and repeating yields the next-largest ones. You buy components only as you need them instead of paying for the full set up front.

3.2.2 The Cost Formula, Corrected

A cautionary tale lives inside this formula. The cost formula was first quoted with the letters tangled — the roles of the feature count and the point count got swapped mid-explanation — and then corrected. The corrected statement is the one above: the exponent 2 sits on the number of features, and the number of training points enters linearly. Doubling your dataset roughly doubles the first term; doubling your feature count quadruples it. Whenever you copy a complexity formula, pin down what each letter counts before you use it.

Rule of thumb from the discussion: once $m$ or $n$ crosses about 500, standard PCA becomes too slow and too memory-hungry to be comfortable. You then want one of the variants below.

3.2.3 Worked Numbers for Image Data

Real-world: these numbers are why photo libraries and medical imaging pipelines never run plain full SVD on raw pixels. A hospital archiving chest X-rays at 2048 by 2048 resolution would face a covariance matrix of over four million squared entries per model refresh; teams there rely on the fast variants covered next, keeping only a few hundred coefficients per scan while retaining diagnostic structure.

3.3 Randomized PCA

Randomized PCA attacks the waste directly: do not compute all $n$ components, compute only the $d$ you will keep, where $d$ is much smaller than $n$. It relies on the principle of randomization. Given the dataset, the method randomly assigns a random subspace of dimension $d$, projects the data onto that subspace, and computes the principal components inside it. Because projections onto a random subspace tend to preserve the geometry of high-variance directions, the top components computed in the subspace closely match the true top components.

3.3.1 The Random Subspace Idea

Why does a random projection work? The leading variance directions of real datasets occupy a tiny fraction of the ambient feature space. Think of a searchlight sweeping a dark warehouse: almost everything interesting stands along a handful of walls, so any random beam catches them. In the same way, a random low-dimensional subspace nearly contains the directions that matter, so the projected data keeps the structure PCA cares about. The dimension of the random subspace is the $d$ you choose, and it replaces $n$ in the cost. Where the analogy bends: an unlucky random subspace can miss a weak-but-important direction, which is why practical implementations sample somewhat more than $d$ dimensions and refine the estimate before trusting it.

3.3.2 The Complexity Drop

Compare with $O(m{n}^2+n^3)$ from the previous section: every occurrence of the huge feature count collapses to the small chosen count.

For deeper detail on how the random projection is constructed and refined, the recommended external lookup is ChatGPT — the explanation there was checked and found accurate — together with the research literature associated with the name Chatterjee.

3.3.3 Running It in scikit-learn

Implementation is a one-parameter change. In scikit-learn you write:

rnd_pca = PCA(n_components=154, svd_solver="randomized")

The value 154 is a user-supplied parameter: it says we want only 154 principal components rather than all of them. The string "randomized" selects the randomized SVD solver.

There is also an automatic mode. If you do not specify a solver, the routine silently switches to the randomized algorithm whenever the larger of the two data counts exceeds 500 and the requested component count is below 80% of the data size. Depending on the size of your data, the library silently moves from the full SVD approach to the randomized approach without being told. This is a smart default for high-dimensional data paired with a large dataset.

Real-world: this is the standard trick when running PCA on image collections or any wide dataset where only a few hundred components survive anyway. Digit-recognition pipelines famously compress 784-pixel handwritten digits to about 150 coefficients while keeping 95% of the variance, which speeds up every downstream classifier trained on the compressed features.

3.4 Incremental PCA

Incremental PCA relaxes the whole-dataset assumption. It exists for two scenarios. First, sometimes the training data is in motion rather than at rest: you never receive all 50,000 records in one shot. Instead, because of how the application works, data arrives over time in batches of, say, 1,000. This is the classic situation with IoT devices, which generate measurements continuously. Second, even if you could collect everything, the full dataset may simply not fit in memory. For memory efficiency you can deliberately split a large dataset into smaller pieces and process them one after another.

3.4.1 Data in Motion: The Streaming Scenario

Streaming data means you cannot wait. An application fed by sensors cannot pause and say "give me all 50,000 points, then I will compute the covariance matrix and run exact PCA." The luxury of waiting does not exist. Whatever learning you do must consume each batch as it lands — streaming data arrives in batches, and waiting for all points is impossible by construction.

3.4.2 The Partial Fit Update Loop

This is an algorithm, not a definition, so read it as a procedure.

In streaming scenarios this is not merely an alternative to ordinary PCA — it is the only available method, because the alternative (collect everything first) is impossible by construction.

Real-world: server monitoring. Suppose you track CPU utilization and RAM utilization of a fleet of servers so you can provision capacity: start new machines, shut machines down, reshape the network layout. Measurements arrive as snapshots every second, or every 500 milliseconds. You run a PCA update on each snapshot batch, refine the components as batches come, and base your resource-management decisions on the current model. There is no choice here: incremental PCA is the only workable design.

3.4.3 Approximation and Per-Batch Cost

Two limitations are worth memorizing.

First, the result is approximate. PCA computed incrementally is less accurate than PCA computed on all data at once. Whether that matters depends entirely on the downstream task that consumes the components. Sometimes the approximation has no noticeable bearing on the task; occasionally it does, and then the honest fix is to get a bigger machine and do exact PCA.

Second, there is real computational overhead, because you re-run a PCA-sized computation for every batch:

Summary judgment: incremental PCA is highly useful for real-time analytics, streaming data processing, and large-scale machine learning where the dataset dwarfs memory. As with randomized PCA, deeper mathematical detail is best chased through ChatGPT and the Chatterjee-linked literature.

3.5 Kernel PCA

Kernel PCA is an extension of PCA using kernel methods, and the motivating problem is classification. Picture a scatter plot with a red class and a blue class curled around each other. No straight line in the original feature space separates all red points from all blue points — the data is not linearly separable. Now run standard PCA on it and reduce dimensions. The PCA-based features inherit the same curl: they are also not linearly separable. So even after reduction, no linear classifier will classify this data perfectly. Standard PCA cannot help downstream linear models here.

3.5.1 Motivation: Classes No Line Can Separate

The fix borrows the kernel idea from SVMs. Apply a transformation $\varphi (x)$ to every data point, where $\varphi$ maps into a much higher-dimensional space. Choose $\varphi$ so that in that high-dimensional space the two classes become linearly separable. Then run PCA in that space; the reduced features now support a downstream linear classifier, which was the whole goal.

Naively this seems suicidal: raising the dimension raises the PCA cost — possibly to millions of dimensions. The kernel trick rescues the idea:

3.5.2 The Kernel Matrix and Its Size

Everything happens through the kernel matrix:

Compare matrix sizes. Ordinary PCA builds its covariance matrix from feature dimensions: that matrix is $n\times n$, scaling with the number of input features. Kernel PCA works instead with the $D\times D$ matrix $K$, where $D$ is the number of data points — the same role $m$ played in earlier sections. Since training sets usually hold several times more instances than features, $D$ typically exceeds $n$. So applying the kernel machinery costs more than plain PCA. But in the non-separable scenario above you could not use plain PCA anyway — the nature of the data forbids it — so kernel PCA is the only route that ends at a linearly separable representation.

3.5.3 The Computation Recipe

The basic recipe has four steps.

Step 3 deserves its algebra in full. Define $1_D$ as the $D\times D$ matrix whose every entry equals $1/D$, and let $H=I-1_D$ be the centering matrix. Multiplying any feature table by $H$ on the left subtracts column means; on the right, row means. Centering the transformed features and rebuilding the Gram matrix gives

$$\begin{array}{rl}{K}^{\prime }& =(H\Phi (X))(H\Phi (X){)}^{\top }\\ & =H\Phi (X)\Phi (X{)}^{\top }H\\ & =HKH\\ & =(I-1_D)K(I-1_D)\\ & =K-1_{D}K-K{1}_D+1_{D}K{1}_D\end{array}$$

where each line follows from the previous one by expanding the product and distributing (line 2 uses associativity; line 5 expands the two binomial products exactly as $(a-b)(c-d)$ expands for scalars). So the intent is precisely to subtract from $K$ two mean-like correction matrices built from entries $1/D$, then add back the grand correction — leaving the transformed features mean-zero in kernel space.

One notation remark connects this to the spoken form: because $K$ built from a symmetric kernel satisfies $K=K^{\top }$, the two middle terms coincide elementwise, and you may meet the compact written form $K^{\prime }=K-21_{D}K+1_{D}K{1}_D$. Both expressions describe the same centered matrix.

Done this way, kernel PCA is quite effective in many domains of modern digital data.

3.5.4 Why Keep the Downstream Classifier Linear

The entire point of the detour is to end at a linear classifier. Why insist? Linear classifiers have real advantages: they run much faster, their results are easier to interpret, and they behave in a stable, reliable way. Their accuracy numbers may not match deep learning, and they use far less processing power. Many applications do not need the extra accuracy boost that deep networks buy, and for those, linear-on-kernel-features is the sweet spot. Know your business and engineering constraints, then pick the correct building block.

Real-world: kernel PCA shows up wherever classes curve around each other — think of sensor layouts or embeddings where a straight boundary is hopeless but a curved one in the original space corresponds to a straight one after the kernel lift.

3.6 Two Applications of PCA-Style Analysis

Two concrete applications show PCA-family tools at work on non-numeric-looking data. Both share one pattern: build a big matrix of items versus attributes, then let PCA expose block structure that no one labeled.

3.6.1 Topic Discovery in News Articles

Modern applications constantly handle textual information, and a common need is to bucketize text snippets into topics. News articles might be sorted into buckets such as sports, politics, culture, movies, or food.

The PCA-flavored route: represent roughly 350 to 400 documents as rows of a document-term matrix, where each column is a term — a bag-of-words feature extracted from the documents (each entry counts how often that word appears in that document; word order is ignored). The size of the vocabulary defines the width of the matrix.

Run PCA on this original document matrix and a banded structure emerges: one band of the decomposition aligns with topic information, the other with document information.

Visual intuition: imagine the matrix printed as a heatmap with documents down the rows and terms across the columns, bright cells for frequent words. After reduction it organizes itself into vertical stripes — a band of columns (words like goal, team, score) lights up together on one set of rows, another band (election, minister, vote) on a different set of rows. Reading off the bands assigns each document to a topic — this set of documents belongs to topic 1, this set to topic 2, then topics 3 and 4, each band holding a fixed-length group. Dimensionality reduction effectively performed topic discovery without anyone labeling a single article.

Real-world: news aggregators and content-management systems use exactly this style of unsupervised bucketing to route articles — the same mathematics also powers recommendation engines that group users by shared item-preference patterns.

3.6.2 Gene Expression Across Experimental Conditions

The same mathematics runs in biology. Transcription-factor and gene-expression studies produce a matrix connecting gene expression levels to experimental conditions. Treat the genes like the terms and the experimental conditions like the documents. For each experimental setting you observe a vector of gene expressions; stacking settings gives the matrix.

PCA-style analysis then finds connections between gene expression patterns and conditions — which genes move together, and under which experimental regimes. Genes that rise and fall together across conditions often participate in the same biological pathway, so the banded structure carries scientific meaning, not just compression.

3.6.3 Where This Module Leads Next

One honest caveat closes the classical portion: PCA is not the most sophisticated dimensionality reduction technique available; many stronger possibilities exist. Right after this module on PCA and its variants, the course moves into deep-neural-network-based feature reduction using autoencoders — the gateway into unsupervised deep learning proper. The classical methods you just saw are the background that makes the deep versions legible.

3.7 Limitations of PCA

PCA earns its popularity honestly, but it has structural limits. Three matter most: its axes are forced to be orthogonal, its statistics are tuned to unimodal clouds, and its linear maps cannot unfold curved data.

3.7.1 Maximum Variance Is Not Always What You Need

By construction, PCA returns eigenvectors of the largest eigenvalues, and those eigenvectors are orthonormal — mutually perpendicular unit vectors. That raises a question: are the maximum-variance dimensions the relevant dimensions for preservation, given all kinds of downstream tasks you might attempt? The answer could be no.

Visualize a dataset shaped so that PCA finds two principal components: one along the axis where data varies most, and a second one perpendicular to it, because the second eigenvector must be orthogonal to the first. For reconstruction purposes this pairing is probably optimal — PCA is built to minimize reconstruction error. But for many downstream tasks, creating the best possible reconstruction is simply not good enough. A direction that maximizes spread can be useless, or even harmful, for separating classes or predicting a target. Imagine two classes stacked thinly along the high-spread axis: PCA keeps that axis happily while flattening away the thin axis that actually tells the classes apart.

The first symptom of this limitation is the blind source separation setting described next — and after it, the multimodality problem.

3.7.2 Unimodal Versus Multimodal Data

Generally speaking, PCA works well when the data is unimodal — one blob, one cluster center, one mode of variation. Picture a single elliptical cloud: PCA axes describe it beautifully.

But if the data distribution has several lobes — multimodal data, with two features $x_1$ and $x_2$ and mass concentrated in two or more separate lumps — PCA tends to perform poorly. Its single global covariance matrix smears the lobes together: one average direction is asked to describe two different crowds at once, and it describes neither well.

Alternative methods exist for this regime:

• Curvilinear component analysis is one — a nonlinear extension treated next.
• Canonical correlation analysis is another relative in this family of alternatives, aimed at relating two views of the same objects rather than at multimodality.

And when the mixture is of sources rather than of spatial lobes, the right tool is independent component analysis, covered fully in the next section.

3.7.3 Curvilinear Component Analysis: The Unfolded Spiral

Curvilinear component analysis is a specific nonlinear extension of PCA. Its promise: preserve the proximity of points in the input space after you transform the data into a new space. In other words, the relative position of data points in the original feature space survives the transformation.

Concrete picture: suppose the data lies on a horseshoe — or, to guess the shape, half spirals — suspended in 3D. You transform it from 3D to 2D by unrolling it, like lifting a coiled spring and pressing it flat onto a sheet of paper.

Formally: if point A is nearer to point B than to point C in the original space — measure the distance, $d(A,B)<d(A,C)$ — then after transformation the same ordering holds. Close-by points map to close-by points; distant points map to distant points. Unfold the spiral-like pattern onto the two-dimensional sheet and you get a roughly rectangular structure: the coil became a stripe. This is a nonlinear projection of the spiral, and the property it protects is called topology preservation — the connected shape of the data survives the change of dimension.

Contrast that with plain PCA, whose straight axes would slice through the coil and destroy exactly these neighborhood relations: two points on adjacent turns of the spiral are physically close in 3D but land on opposite ends of PCA's flattened view. When data is distributed in a nonlinear manner, neighborhood-preserving nonlinear projections beat variance-maximizing linear ones.

3.8 Independent Component Analysis

Independent component analysis (ICA) answers the weakness PCA showed with mixed sources. Where PCA projects combined data onto orthogonal axes, ICA finds independent dimensions — non-orthogonal coordinates that align with the true sources hiding inside the mixture.

3.8.1 The Cocktail Party Problem

The flagship application has a popular nickname: the cocktail party problem, technically called blind source separation (BSS). Imagine a room with several uncorrelated sources of sound: one person talking, another singing, a television playing. A microphone in that room catches everything at once. What the microphone records is simply a combination of all these uncorrelated signals mashed together, and from that single jumbled channel you cannot make sense of who said what.

Employing PCA here does not recover the separate audio sources — PCA has no ability to pull independent sources out of the jumbled mixture. Its orthogonal axes blend the sources instead of isolating them. A different type of analysis makes sense: independent component analysis.

The physical picture: several sound sources sit in a room, and several microphones listen. Each microphone records its own mixture. Because each source stands at a different distance from each microphone, the mixing weights differ from microphone to microphone — the weights $a_{11},a_{12},a_{13},a_{14}$ for the first microphone differ from $a_{21},a_{22},a_{23},a_{24}$ for the second, and so on for every microphone.

3.8.2 The Linear Mixing Model

Write the physics as algebra. Let $S$ collect the unknown source signals and $X$ collect the microphone recordings:

The difficulty: in this equation you know neither $A$ nor $S$. The only thing you observe is $X$. Applying ICA to $X$ estimates the sources anyway — and that recovery, performed with no knowledge of the mixing matrix, is exactly why the problem is called blind source separation.

A compact two-dimensional formalism makes the structure vivid. Let $x=(x_1,x_2{)}^{\top }$ be an observed two-dimensional vector formed from two signal sources:

$$x=a_1{s}_1+a_2{s}_2=\left(\begin{array}{cc}{a}_{11}& a_{12}\\ a_{21}& a_{22}\end{array}\right)\left(\begin{array}{c}{s}_1\\ s_2\end{array}\right)$$

Read this as a basis expansion: $a_1$ and $a_2$ act as the weighting vectors — the basis vectors — and $s_1,s_2$ act as the basis coefficients. The whole of ICA rests on one statistical claim about those coefficients, taken up next.

3.8.3 What Independence Requires

The method assumes $s_1,s_2,s_3,s_4$ — the sources — are statistically independent of one another: each is generated on its own, with no connection between any pair. Knowing one source tells you nothing about another.

Independence is a real restriction, and a musical counterexample shows its edge. Someone singing while music accompanies the song is not independence — the two tracks are coordinated, each following the other's tempo and key. But one person singing while another person recites a poem, with no coordination between them, is genuine independence — and there ICA produces real results.

Geometrically, contrast the two methods on mixed two-source data. PCA, handed data containing two modes, yields two orthogonal axes: call them $E_1$ and $E_2$, at 90 degrees, so their dot product is zero — but each axis carries a blend of both sources. ICA instead identifies the axis of maximum variance separately for each independent source. Instead of projecting combined data onto two orthogonal axes, you project onto two independent axes — generally not perpendicular at all.

3.8.4 Applications Beyond Sound

"Source" never meant only sound. ICA has found many other applications:

• EEG brain signals. Electroencephalography places many sensors on the head; each measures brain activity near its position, and the readings combine into the recorded signal. Running ICA on that extracted signal recovers the underlying components of brain activity. Brain-imaging modalities beyond EEG serve as inputs too.
• Medical signal processing generally.
• Topic extraction from documents.
• Finding hidden factors in financial data — econometrics, time series analysis, and stock market data.
• Data clustering.
• Images, including face representations discussed below.

3.8.5 How the Sources Are Recovered

Several estimation families exist. The standard route is maximum likelihood based learning, which maximizes the probability of finding the sources $s_1$ and $s_2$ given the observations. Alternatives include entropy maximization and minimum entropy coding. Each family differs in the statistic it squeezes, but all exploit the same advantage: independent sources have distributions whose higher-order structure betrays the mixing angles. The exact mathematics of each method deserves its own study; the headline is that ICA is a powerful, mature technique.

3.8.6 Eigenfaces versus Factorial Faces

Face imagery supplies a beautiful worked comparison. The classical method is eigenfaces: given a dataset of mug shots, compute the PCA of the image set; the eigenfaces are the images associated with the maximum eigenvalues — ghostly average-face-like pictures capturing the biggest variations. You can do useful things with eigenfaces, but some things work better with factorial faces: instead of hunting principal components of the face set, you model the probability distribution of face images directly, using factorial representation techniques of the ICA type. The face distribution gets factorized into independent factors.

Add one more twist, and it teaches the deep-learning lesson. Replace the nearest neighbor classifier with a multilayer perceptron (MLP) and the relative benefit of factorial faces shrinks. Why? A multilayer perceptron has many hidden layers and many hidden nodes, so plenty of feature extraction happens inside the classifier itself. That is why in deep learning we say carefully crafting features is not that critical compared to traditional classifiers such as nearest neighbor or logistic regression. The relative gain from ICA-based factorial faces diminishes as the downstream classifier grows more complex — but an MLP brings disadvantages of its own, so nothing is free. These tools exist so that, depending on business and engineering constraints, you can pick the right building block for your solution.

3.8.7 PCA versus ICA at a Glance

Set the two side by side.

PCA is useful for compression and can serve classification. It is a second-order statistics method, because the covariance matrix is a second-order statistic, and its components are orthogonal. Second-order statistical structure is richest for uncorrelated, Gaussian components — which connects to the unimodal story from the previous section. ICA targets independent, non-Gaussian components and exploits higher-order statistics, and its recovered dimensions generally will not be orthogonal. Pick according to whether your data hides uncorrelated blobs or genuinely distinct sources.

One housekeeping note rounds out the topic. The broader reference deck lists a few further techniques: canonical correlation analysis (CCA), which relates two sets of variables, and nonlinear embedding methods — t-SNE is the embedding technique meant here. None of these are part of the examined material; they are optional self-study.

3.9 From Supervised to Unsupervised Learning

With the classical toolkit complete, reset the frame. Everything so far ran without labels. Time to place that in contrast with what deep networks usually do — and to preview where the course goes.

3.9.1 Supervised Learning in One Paragraph

In supervised learning you have the original data $X$ and the training labels. The goal is to realize $y=f(x)$: train a deep network so its weights and activations map input $X$ to outcome $y$.

Familiar examples:

• Object recognition — given an image, answer "dog", "car", "truck", or "umbrella".
• Object detection in a general scene — attach a bounding box around each object and learn the coordinates of those boxes, together with classifying the pixels enclosed by each box into one class or another; that detection setting gets fuller treatment in a computer vision course.
• Image classification with CNNs and their many architectures — the canonical supervised example.

Interestingly, a CNN can also run in an unsupervised setting, discussed a little later.

3.9.2 What Works Without Labels

Unsupervised learning has no access to labels, yet plenty of useful work gets done from the data alone:

• Find dimensions of maximum variation — PCA, as practiced all through this module.
• Separate mixed sources — ICA solving blind source separation without a single label.
• Group points into distinct clusters — k-means clustering, which you met in the machine learning course.
• Fit soft clusters — Gaussian mixture models find cluster structure without knowing any training label.
• Learn features — discover the main highlights that recur across a large image collection. That is feature learning, also called representation learning.

Notice the pattern: every task on this list extracts structure that labels would normally provide — group membership, source identity, informative axes — directly from the data itself.

3.9.3 The Road Ahead

Feature learning is the immediate destination: the autoencoder. After that comes density estimation — estimating the probability density of the data, which then enables creating new data via sampling from the estimated densities. From here on, perhaps 90% or more of the course is devoted to feature learning and density estimation.

3.10 The Autoencoder Architecture

An autoencoder is an unsupervised deep learning method for feature extraction. Given original data, it learns a lower-dimensional representation called the code. The name tells the story: it encodes the data — creates codes out of it, a feature representation of the original — and the codes are often much lower-dimensional than the input. PCA also did dimensionality reduction; here the same goal is pursued with the framework of deep learning, and we will see that autoencoders do this encoding much better than PCA.

3.10.1 Train It to Copy Its Input

The basic principle of training is delightfully strange: train the network so the output copies the input, up to some transformations. The output should come out close to equal to the input — in other words, use the input itself as a proxy for the target outcomes. No labels exist, so the input plays their role.

Concretely, the general framework looks like this: original data dimensions $x_1$ through $x_5$ — a five-dimensional input — each connected to three hidden nodes, and the hidden nodes' outputs connected onward to output nodes through another tier of weights.

The loss function makes it precise:

Once training converges, you own a trained autoencoder, and the activations of the hidden layer are the so-called features — reduced-dimensional data, the reduced-dimensional encoding of the input.

An important freedom: these hidden layers can use nonlinear activations, such as sigmoid or ReLU. And here is the payoff — with nonlinearities in place, the features you can extract have much higher fidelity than PCA's, for the same number of dimensions. Fix the code size equal to the PCA component count, and the autoencoder's ability to reconstruct beats the PCA pipeline. The diagram above uses just one hidden layer; a typical autoencoder stacks several, with the final hidden layers delivering the features.

3.10.2 Encoder, Decoder, and Code

Write the two halves as equations:

Names for the halves: the input-to-hidden part is the encoder layer; the hidden-to-output part is the decoder. You encode the original data so that decoding it reconstructs something very close to the original. The decoder's only job is to train the encoder: once training is over, you throw away the decoder — and the loss function too — and keep just the encoder. From then on, fresh data flows in and the hidden activations $h$ come out as its code.

The full system has three parts: an encoder network, also called the recognition network, running from input through possibly several hidden layers; a decoder, also called the generative network, producing the reconstruction $\hat{x}=g(f(x))$; and a loss function — for continuous-valued signals, typically the squared L2 norm. After training, deploy only the encoder, and optionally feed its codes to a classical machine learning algorithm downstream, if the business requires only that level of performance.

Visual intuition: sketch the network as an hourglass. Data enters at full width on the left (five nodes), squeezes through the narrow waist (the three-node code), and expands back to full width on the right (five reconstructed nodes). The waist is where the treasure sits: everything the network managed to keep about the input had to fit through those few nodes.

3.10.3 Undercomplete Autoencoders

Typically the dimension of $h$ is much smaller than the dimension of $x$. That configuration is the undercomplete autoencoder. Fewer hidden nodes than inputs forces information loss — the reconstruction is inherently lossy, never perfect.

But there is a silver lining with real diagnostic power: if the reconstruction still comes out at very high quality — if that squared L2 number comes out small — then the hidden activations must have captured the important characteristics of the data. Had they failed to capture the important features, high-fidelity reconstruction from fewer numbers would be impossible. So in the undercomplete setting, reconstruction quality doubles as a certificate that the code is meaningful.

3.10.4 Matching Activations to Input Type

Which activation functions belong where? The answer depends on the input's value range, and getting it wrong breaks training.

Case one: continuous real-valued input, ranging from minus infinity to plus infinity. The output layer should emit real values on the same range, so use a linear activation at the output. The hidden layers may use nonlinear functions — sigmoid, ReLU, and friends.

Case two: binary input, strings like 01101 or 1111 or 000. During training you want the output close to the same binary string. The output activation must guarantee values between 0 and 1, and the sigmoid — the logistic function — does exactly that. Hidden layers suit tanh or ReLU.

Now a quick decision drill that cements the logic: binary input, three candidate output activations — linear, tanh, logistic.

• Tanh outputs values between $-1$ and $+1$: rejected, because it can emit $-1$, which no binary target ever takes.
• Linear outputs range over all reals: rejected, because it can overshoot past 1.
• Logistic alone confines itself to $[0,1]$, matching binary inputs. Winner: logistic.

For real-valued input the choices flip: linear at the output is right, with ReLU or sigmoid options in the hidden layers.

During training for the real-valued case, the goal is to find $W$, $W^{\prime }$, $b$, and $c$ — weights input-to-hidden, weights hidden-to-output, and the two bias vectors — minimizing the reconstruction error summed over all training patterns. The whole objective can always be written in vector form.

Real-world: anomaly detection in machine fleets works exactly this way. Train an undercomplete autoencoder on normal vibration sensor readings from a wind turbine gearbox; when new readings reconstruct poorly, the code did not capture them — which is precisely the signature of a fault no label ever described.

3.11 Training Autoencoders: Loss Functions and Backpropagation

3.11.1 Forward Pass and Vector Notation

All the training mathematics is standard backpropagation. The forward pass computes a total input to the hidden layer, then squashes it:

Note that ${\mathbf{a}}_1$ and ${\mathbf{h}}_1$ are both vectors. That observation motivates the notation convention used throughout: bold lowercase letters denote vectors ($\mathbf{x},\mathbf{b},{\mathbf{h}}_1$), bold uppercase letters denote matrices ($\mathbf{W}$), and non-bold letters denote scalars. Keep that convention in mind for every formula ahead.

3.11.2 Binary Cross-Entropy for Binary Inputs

Switch the input to binary strings. What loss function compares one binary string with another effectively? The answer is binary cross-entropy — and because an autoencoder has many output units, each independently 0 or 1, you use a sum of binary cross-entropies: one term per output unit, accumulated over the outputs $j=1$ to $N$ and over all training patterns. Minimize that sum with respect to the $W$'s, $b$'s, and $c$'s.

The bridge back is the logistic regression loss you already know:

Watch the two cases work. If $o=1$, the second term vanishes and $L=-\log \hat{y}$: the loss shrinks only as $\hat{y}$ climbs toward 1. If $o=0$, the first term vanishes and $L=-\log (1-\hat{y})$: the loss shrinks only as $\hat{y}$ falls toward 0. Numbers make it vivid — target $o=1$ with prediction $\hat{y}=0.9$ costs $-\log (0.9)\approx 0.105$; the same prediction against $o=0$ costs $-\log (0.1)\approx 2.303$, about twenty-two times worse. Exactly this per-unit loss, summed across units, is what the autoencoder minimizes for binary inputs.

3.11.3 Multi-class Versus Multi-output Classification

The longest, most valuable stretch of this session is a guided dialogue that separates two easily-confused classification settings. It proceeded as a chain of questions and answers.

3.11.4 Backpropagation and the Exam

Every autoencoder variant discussed so far trains with standard backpropagation. Recall what backpropagation fundamentally is: two ingredients — the chain rule of differentiation, together with total derivatives — combined to push error signals backward through the network and derive every weight-update formula. Those two together let you derive all the update equations for these networks. The published update formulas are nothing more than the standard backpropagation algorithm applied to this architecture; verifying them yourself is worthwhile review.

3.12 When a Linear Autoencoder Equals PCA

Autoencoders can give you components very similar to PCA's principal components — under certain restrictive conditions. There are three: how you preprocess the data, which activations you allow, and which loss you pick.

3.12.1 Mean-Normalize First

Condition one concerns preprocessing. Take the data matrix with $m$ training instances and $n$ features. First compute the component-wise mean vector: for each feature column, sum the values over all training points $i=1$ to $m$ and divide by $m$:

$$\mu =\frac{1}{m}\sum _{i=1}^m{x}^{(i)}$$

Subtract $\mu$ from every training instance. Then divide by $\sqrt{m}$, the square root of the number of data points.

Why that divisor? Recall what the covariance matrix is: squared deviations averaged over the $m$ points,

$$C=\frac{1}{m}(X-1{\mu }^{\top }{)}^{\top }(X-1{\mu }^{\top })$$

where $X\in {\mathbb{R}}^{m\times n}$ stacks training instances as rows and $1\in {\mathbb{R}}^m$ is the all-ones vector, so $X-1{\mu }^{\top }$ subtracts $\mu$ from every row. Now watch what scaling the data itself by $1/\sqrt{m}$ does. Let $Z=(X-1{\mu }^{\top })/\sqrt{m}$. Then

$$Z^{\top }Z=\frac{1}{\sqrt{m}}(X-1{\mu }^{\top }{)}^{\top }\cdot \frac{1}{\sqrt{m}}(X-1{\mu }^{\top })=\frac{1}{m}(X-1{\mu }^{\top }{)}^{\top }(X-1{\mu }^{\top })=C$$

The two factors of $1/\sqrt{m}$ multiply into the single factor $1/m$ that the covariance formula carries. So dividing by $\sqrt{m}$ bakes the covariance's own normalization into the data, leaving the processed input mean-normalized with its second-order statistics equal to the covariance used in the PCA calculation — exactly the stated goal. The spoken rule ("divide by the square root of the number of data points") and the covariance identity check out against each other.

3.12.2 The Equivalence Result

Conditions two and three concern the network: use a linear encoder and a linear decoder — which simply means linear activation functions in the hidden layer and the output layer — and a squared-error loss function.

Flip the viewpoint and it sounds even cleaner: PCA is a special case of the autoencoder — the special case you get when you use a linear encoder, a linear decoder, and squared loss on mean-normalized data. So this configuration offers no practical benefit over PCA. Nothing new is learned.

3.12.3 What Depth and Nonlinearity Buy

Autoencoders earn their advantage over PCA precisely by breaking the restrictive conditions: stack multiple hidden layers in the encoder and the decoder, and use nonlinear activation functions in the hidden and output layers. Nonlinearity ensures the computed features are no longer linear combinations of the inputs — not the linear features PCA is stuck with. Depth lets those nonlinear features compose.

A second freedom follows: PCA's components must be mutually orthogonal, but an autoencoder's weights carry no orthogonality requirement — that constraint only reappears under the special linear-plus-squared-loss regime. Free of it, autoencoders deliver general, potentially nonlinear projections whose ability to reconstruct is far superior to PCA-family methods.

3.13 Overcomplete Autoencoders and the Identity Trap

So far the hidden layer shrank the data. Choose the number of hidden nodes equal to the number of inputs, or greater, and you have an overcomplete autoencoder — and taken naively it is a trap.

3.13.1 The Identity Mapping Walkthrough

With enough hidden units the network can drive reconstruction error to zero — while recovering nothing interesting at all. It simply copies the input. Watch it happen in a minimal worked example.

Setup: a two-dimensional input with components $x_1$ and $x_2$; a hidden layer with two units; linear activation functions throughout; and all biases set to zero, $b=0$ and $c=0$. Now choose the weight matrices to be the identity:

$$W=\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right),W^{\prime }=\left(\begin{array}{cc}1& 0\\ 0& 1\end{array}\right)$$

Nothing useful was found; the network is just copying. And the identity matrices are only the cheapest culprit: any invertible $W$ paired with $W^{\prime }=W^{-1}$ copies every input exactly while shuffling coordinates pointlessly. With dimensions of the hidden layer greater than or equal to the input dimensions and no other constraints, you get nonsense — no sensible feature representation at all.

3.13.2 Regularization Changes the Story

Why spend time on this nonsensical configuration? Because the overcomplete representation becomes genuinely useful as soon as you put constraints on the architecture — and in the autoencoder world those constraints go by the name regularization:

Constrain the spare capacity, and it gets spent on structure instead of copying.

Recall regularization from the deep learning course, where it appeared in a specific role: controlling overfitting. L2 regularization stops the network from over-learning, protecting validation error. L1 regularization acted as a feature selection mechanism, pushing unimportant weights to exactly zero. The same instruments, attached to overcomplete autoencoders, unlock the useful variants of the next section — sparsity of representation chief among them.

3.14 Sparse and Contractive Autoencoders

The identity trap showed that unconstrained capacity learns nothing. The variants in this section all fix that the same way: constrain what the hidden layer may do, and structure has nowhere to hide but in the code.

3.14.1 Sparse Codes and Dictionary Learning

Sparsity of representation is a constraint demanding that, out of a large population of inputs, each hidden unit fires for only a small fraction of input patterns — responding to perhaps 10% of them — and stays at zero for the remaining 90%. The hidden representation as a whole becomes highly sparse: for any given input, only a few hidden units are active.

Effectively, you encode the original data as if into binary strings — mostly-off switch patterns. The family of methods built on this idea travels under the name of sparse autoencoders, and the underlying representation principle is known in the literature as dictionary learning: the network's weights become a dictionary of reusable atoms, and each input is described by combining just a few entries from it — like writing any English sentence using only a handful of words picked fresh from a large dictionary.

Feel the compression concretely:

3.14.2 Contractive Autoencoders

A second constrained variant is the contractive autoencoder. Its promise: unless the input changes a lot, the code does not change much. The encoding function contracts small wiggles in the input space, so the learned representation stays stable under minor perturbations of the data — sensor jitter, compression artifacts, small lighting shifts. Where sparsity constrains which units may respond, contraction constrains how strongly the code may react.

3.14.3 Denoising and Missing Inputs: A Preview

The same regularization machinery serves a third purpose: removing noise, or coping with missing inputs, in the original data — the territory of the denoising autoencoder, which trains on corrupted inputs but demands clean reconstructions, forcing the code to look past the noise.

Time ran out before covering these constraint types in full, so they lead the agenda of the following session, alongside a deep autoencoder architecture: the convolutional autoencoder, previewed next.

3.15 Convolutional Autoencoders

The convolutional autoencoder takes a CNN-type architecture and uses it in a novel way: no training labels anywhere. The convolutional layers exist to extract features of the original data — in autoencoder language, to create an encoded version of the input.

3.15.1 The Encoder Path

The encoder path stacks convolutional layers interleaved with max pooling. Each pooling stage shrinks the spatial size, so you can watch the representation becoming smaller and smaller: 160 wide, then 80, then 40, then 20. Eventually the encoded dimension at the bottleneck is much smaller than the original input dimension. That bottleneck is the code, extracted purely by convolutions — each stage keeping what its filters found important and pooling away the rest.

3.15.2 The Decoder Path and Transpose Convolution

The decoder reverses the journey, and here is the novel piece: instead of standard convolutions it uses transpose convolution — the operation exposed in libraries under names like Conv2DTranspose. Recall that transpose convolution was taught in the deep neural network course, though not exercised there.

Its special power comes from the fractional stride: the output feature map can come out bigger than the input. Two equivalent ways to picture it, both useful:

• Stretch first, convolve second: insert empty rows and columns of zeros between the input's cells, then run an ordinary convolution over the stretched map.
• Or think of a normal convolution whose stride is a fraction, say $1/2$: the filter steps half as far per output cell, so outputs multiply.

Either way, in a transposed convolution layer the stride defines how much the input stretches — the larger the stride, the larger the output, exactly opposite to ordinary convolution or pooling.

3.15.3 Training Without Labels

Training needs no labels, because the target is the input itself.

That completes the convolutional autoencoder: convolution layers as encoder, transpose-convolution layers as decoder, squared difference as the training signal.

Real-world: this exact encoder-decoder shape powers image denoising and compression pipelines in medical imaging, where clean reconstructions from noisy scans matter more than class labels, and it is the architectural ancestor of the segmentation networks used throughout computer vision.

Exam Guidance Summary

• Format and scope. The exam is closed book. Everything asked will come from covered material — the scope rule is simple: no question will come from anything not covered in the sessions. Covering the PCA variants and ICA discussed here is enough preparation for this block.
• Autoencoder backpropagation. Expect a problem requiring you to apply the backpropagation algorithm to compute weight updates for an autoencoder network, iteration by iteration. The update formulas are standard backpropagation — chain rule plus total derivatives. Practice the differentiation by hand until it is routine; after the prerequisite courses it should not be a big deal.
• Notation discipline. When writing complexity formulas, pin down each letter before using it: the exponent sits on the feature count, the point count enters linearly. The cost correction from this session is the cautionary example to remember.
• Loss-function selection. Be ready to choose the right loss and output activation for an input type: squared error with linear output for real-valued data; summed binary cross-entropy with sigmoid outputs for binary strings; softmax plus plain cross-entropy only when targets are one-hot multi-class.
• Optional reading. Canonical correlation analysis, t-SNE-like embedding techniques, and the illustrative PCA calculations at the back of the reference deck are outside the examined scope — the calculations are review from earlier coursework, worth a skim at most.

Key Industry Applications

• Server and cloud capacity management. Incremental PCA on CPU-utilization and RAM-utilization snapshots arriving every 500 milliseconds to 1 second supports resource-management decisions: when to start new machines, when to turn machines off, how to provision the network architecture.
• IoT sensor streams. Devices emitting data in batches of about 1,000 records — never all 50,000 at once — are the natural habitat of incremental PCA's partial-fit loop.
• Large-scale memory-bound analytics. When a dataset cannot fit in memory, splitting it into pieces and running incremental PCA trades a tolerable approximation for feasibility.
• News and document routing. Bag-of-words matrices over roughly 350 to 400 documents, reduced by PCA, expose banded topic structure for sorting articles into sports, politics, culture, movies, and food buckets.
• Genomics. Gene-expression matrices analyzed across experimental conditions — genes playing the role of terms, conditions the role of documents — reveal expression-condition connections relevant to transcription-factor studies.
• Neuroscience and medical signal processing. ICA decomposes EEG recordings from head-mounted sensors into underlying brain-activity components; medical signal processing broadly adopts the same tool.
• Finance and econometrics. ICA finds hidden factors in financial data and supports time series analysis of stock market data.
• Face recognition systems. The eigenfaces-versus-factorial-faces comparison — 22% versus 91% accuracy at 20 dimensions with a nearest neighbor classifier — guides representation choice in biometric pipelines.
• Production ML libraries. scikit-learn exposes randomized PCA through svd_solver="randomized" with a user-chosen n_components such as 154, and auto-selects the randomized path whenever the larger data dimension exceeds 500 and the requested component count stays below 80% of the data size — the default behind many production dimensionality-reduction jobs.