Regularized Autoencoders and Likelihood-Based Generative Models

# Regularized Autoencoders and Likelihood-Based Generative Models

4.1 Recap: Undercomplete Autoencoders and Lossy Encoding

An autoencoder is a pair of networks trained together. The encoder squeezes the input into a small code, and the decoder rebuilds the input from that code. In symbols, for an input $x\in {\mathbb{R}}^d$ (a vector of $d$ numbers), a hidden code $h\in {\mathbb{R}}^{d_h}$, and weights $W\in {\mathbb{R}}^{d\times d_h}$ with bias $b\in {\mathbb{R}}^{d_h}$:

$$h=g\left(W^{\top }x+b\right),\hat{x}=g^{\ast }\left(W^{\ast \top }h+c\right)$$

Here $g$ is the hidden activation function of the encoder, $g^{\ast }$ the activation of the decoder output layer, $W^{\ast }\in {\mathbb{R}}^{d_h\times d}$ and $c\in {\mathbb{R}}^d$ the decoder's weight matrix and bias, and $\hat{x}$ the reconstruction. Both halves train together with backpropagation, and the training signal is just the difference between $x$ and $\hat{x}$.

In the undercomplete formulation introduced in the earlier session, the hidden layer has fewer nodes than the input layer: $d_h<d$. Because the code lives in a smaller-dimensional space than the original data, it can hold only some of the main underlying factors that describe the dataset.

This narrowing has a name. Projecting the original space onto the smaller space spanned by the encoder nodes always loses some information, no matter how little. That is why the undercomplete autoencoder performs a lossy encoding: when the code passes through the decoder, the reconstruction can land very close to the original data, but some loss always remains.

A concrete picture helps. Think of retelling a two-hour film as a one-minute spoken summary. You cannot keep every scene; you must pick which threads carry the story. A good summary loses detail but keeps the plot. The autoencoder does the same job with gradients instead of judgment: the bottleneck forces it to decide which directions of variation in the data matter most. The analogy has a limit — a person chooses what matters on purpose, while the network discovers it mechanically by minimizing reconstruction error.

The quality bar for any autoencoder architecture follows from this: an architecture counts as better if it reconstructs the data well while using a smaller-dimensional subspace spanned by the hidden nodes. Reconstruction quality alone proves nothing; quality per code dimension is the honest measure.

4.1.1 Input Types Constrain Activations and Losses

The nature of the data being auto-encoded restricts the design choices before training even starts.

For real, continuous inputs, the output layer needs one kind of activation and one kind of loss; for binary inputs, different choices fit better. The same restriction applies to the activations inside the encoding nodes. So before building any autoencoder, ask what the input looks like — the answer decides the output activation, the output loss, and the hidden activations.

4.1.2 Backpropagation Assumptions and Exam Expectations

Training an autoencoder uses straightforward backpropagation, and every weight update formula in this course follows from it. The chain rule carries the reconstruction error backwards through the decoder, through the code layer, and into the encoder; each weight receives the share of the blame it earned in the forward pass.

4.2 Overcomplete Autoencoders Need Regularization

The mirror image of the undercomplete case also exists. An overcomplete autoencoder has as many hidden nodes as the input dimension, or even more ($d_h\ge d$). Without any extra adjustment, such a network learns nothing worth knowing: the original data gets copied through the connections to the hidden nodes and out to the output.

4.2.1 What Goes Wrong Without Constraints

Think about what copying means mechanically. Every input value has a private route to the output: a weight of about 1 on its own path and about 0 everywhere else turns the whole stack into the identity map. The network can then score zero reconstruction error without ever compressing, abstracting, or noticing structure. No latent variation is forced to emerge, because none is needed.

The copy behavior destroys exactly what we wanted from the bottleneck — insight into the latent variations within the data and the deep underlying features on which the whole dataset rests. With as many code dimensions as input dimensions, that discovery capability disappears. A useful analogy: hand a student the answer key before an exam and ask them to "solve" the paper. Perfect score, zero understanding. The overcomplete network holds the answer key — the input itself — so it never has to learn anything about the subject. The analogy breaks only in that the network does not even know it is copying; gradient descent simply never encounters pressure to do otherwise.

When to pick which? An unregularized overcomplete autoencoder is never the goal; it becomes interesting only once a penalty forces the code to earn its size.

4.2.2 Three Autoencoder-Specific Penalties

Overcomplete autoencoders become useful when regularization enters the training process, which gives the family its name: regularized autoencoders. Besides the standard L1 and L2 penalties learned in earlier machine learning and deep neural network courses, three penalties are specific to autoencoders:

• Sparsity of representation — force most coding units to stay silent most of the time. This produces sparse autoencoders.
• Robustness to noise or missing inputs — train on corrupted data but demand clean reconstructions. This produces denoising autoencoders.
• Smallness of derivatives of the representation — make the coded values change slowly with the inputs. This produces contractive autoencoders.

Each penalty attacks the copy shortcut differently. Sparsity makes a full copy expensive, because the copy would light up every coding unit. Noise robustness breaks the private route, because the input that arrives is damaged while the target stays clean. Small derivatives make the code blind to small wiggles, so passing raw values through earns nothing. This lecture covers all four ideas in turn: L1/L2 next, then tied weights and the contractive variant, then denoising and sparsity in detail.

4.3 L1 and L2 Regularized Autoencoders

Collect every weight in the architecture into one symbol $\theta$. This includes all entries of $W$, the weight matrix from the input layer to the encoder layer, and all entries of $W^{\ast }$, the weight matrix from the encoder layer to the decoder (output) layer. Two bias vectors also exist — $b$ at the encoder side and $c$ at the decoder side — and they play a special role explained below. In this section $\lambda$ (lambda) is the regularization constant deciding how strongly weights get punished for growing. Watch out later: the same Greek letter plays the role of a step size in Section 4.4, because the lecture used it there; the meaning is announced wherever it appears.

4.3.1 The L2 Objective and Its Symbols

For real-valued inputs, the plain training goal is to minimize the squared reconstruction error. Stated in words from class: minimize this reconstruction error, assuming real data whose range lies between minus infinity and plus infinity, together with the L2 norm of theta, squared. Reconstructed, the objective reads:

$${\mathcal{L}}_{L2}(\theta )=\sum _{i=1}^m\Vert x^{(i)}-{\hat{x}}^{(i)}{\Vert }_2^2+\lambda \Vert \theta {\Vert }_2^2$$

where $x^{(i)}$ is the i-th training input, ${\hat{x}}^{(i)}$ is its reconstruction, $m$ is the number of training points, and $\lambda >0$ balances the two goals. The penalty $\Vert \theta {\Vert }_2^2$ sums the squares of every weight in both matrices. Its effect is called weight decay: all weights feel pressure to shrink.

Why does squaring produce shrinking? Differentiate the objective with respect to any single weight $w$. The reconstruction part contributes whatever the data demands, call it $\partial R/\partial w$, and the penalty contributes $2\lambda w$:

$$\frac{\partial {\mathcal{L}}_{L2}}{\partial w}=\frac{\partial R}{\partial w}+2\lambda w$$

A gradient-descent step with learning rate $\eta$ then gives:

$$w\leftarrow w-\eta \left(\frac{\partial R}{\partial w}+2\lambda w\right)=(1-2\lambda \eta )w-\eta \frac{\partial R}{\partial w}$$

Read the middle expression carefully: before the data gradient does anything, every weight is multiplied by the factor $(1-2\lambda \eta )$, a number just below one. Each training step shaves a fixed percentage off every weight. A weight the data never uses therefore decays toward zero geometrically.

Notation note: some textbooks write this penalty with a factor of one half, $(\lambda /2)\Vert \theta {\Vert }_2^2$, so its derivative is $\lambda w$ instead of $2\lambda w$. The difference is absorbed by re-scaling $\lambda$; the behavior is identical.

Training runs by backpropagation over the whole objective, updating only the weights. One carry-over rule from earlier ML and deep learning courses matters here: apply regularization only to the weights, never to the biases $b$ and $c$. The biases stay outside the penalty term. The reason is capacity: a bias merely shifts a unit's response left or right, it does not multiply the influence of any input direction, so growing biases does not create the kind of flexibility that overfits. Penalizing them would mainly cost fitting power while buying almost nothing.

4.3.2 The L1 Objective and Feature Selection

The L1 variant swaps the squared penalty for absolute values. In words: use lambda times the sum over theta of the absolute values of theta; this has a tendency to push the thetas toward zero. Formally:

$${\mathcal{L}}_{L1}(\theta )=\sum _{i=1}^m\Vert x^{(i)}-{\hat{x}}^{(i)}{\Vert }_2^2+\lambda \sum _{w\in \theta }|w|=\cdots +\lambda \Vert \theta {\Vert }_1$$

This is the lasso idea from earlier coursework. The mechanical difference shows up in the derivative. The absolute-value function $|w|$ has slope $-1$ for negative $w$ and $+1$ for positive $w$, written $\mathrm{sign}(w)$. So the update becomes:

$$w\leftarrow w-\eta \left(\frac{\partial R}{\partial w}+\lambda \mathrm{sign}(w)\right)$$

Every step drags each weight toward zero by the same fixed amount $\eta \lambda$, whether the weight is large or tiny. Small weights get pushed all the way past zero and stop there, instead of lingering at 0.001 forever as they do under L2's proportional pull.

Because lasso drives some weights all the way to zero, it can select the more relevant subset of features out of an original feature set containing irrelevant entries — feature selection. Both penalties apply to overcomplete autoencoders and equally to undercomplete ones; in both settings they provide better generalization.

When to pick which? Use L2 when you believe many features each contribute a little; use L1 when you suspect only a few input directions matter at all.

4.3.3 Practical Notes on Regularization

Real-world: for extra problem practice, ChatGPT turns out to be effective at generating questions in this exact area, and its solutions are mostly correct.

4.4 Tied Weights

A special structural constraint can act as a regularizer: force the decoder weight matrix to be the transpose of the encoder weight matrix,

$$W^{\ast }=W^{\top }$$

A concrete shape example from class: suppose the encoder maps a 4-dimensional input to 12 hidden units, so $W$ is a $4\times 12$ matrix (and the code is $h=g(W^{\top }x+b)$, where $W^{\top }x\in {\mathbb{R}}^{12}$); then the decoder mapping from those 12 units back down to 4 outputs, called $W^{\ast }$, is a $12\times 4$ matrix — precisely the transpose of $W$. Entry $(j,i)$ of $W^{\ast }$ always equals entry $(i,j)$ of $W$: the strength of the connection from input $i$ into hidden unit $j$ is declared, by fiat, equal to the strength of the connection from hidden unit $j$ back out to output $i$.

4.4.1 Why Plain Gradients Break the Constraint

If you run ordinary updates on both matrices, the constraint breaks immediately. Stated in words from class: for calculating the partial of the loss with respect to $W^{\ast }$, only the output-side activation functions are used; but for $W$, information flows through both layers.

Concretely, $\partial L/\partial W^{\ast }$ is read off directly at the decoder output layer, while $\partial L/\partial W$ must travel the full chain rule path: loss, then decoder, then through the hidden activations, before reaching the encoder weights. The two gradients are therefore different numbers in general. After one plain step,

$$W_{current}^{\ast }-\eta \frac{\partial L}{\partial W^{\ast }}\ne {\left(W_{current}-\eta \frac{\partial L}{\partial W}\right)}^{\top }$$

so the freshly updated matrices stop being transposes of each other, and whatever benefit tying promised evaporates from iteration two onward.

4.4.2 Synchronized Updates Step by Step

The fix is a two-phase update computed every iteration. First compute temporary, unconstrained updates for both matrices:

$$W_{temp}^{\ast }=W_{current}^{\ast }-\lambda \frac{\partial L}{\partial W^{\ast }},W_{temp}=W_{current}-\lambda \frac{\partial L}{\partial W}$$

Here $\lambda$ plays the role of the step size in the update rule (what other texts often call $\eta$) — not the regularization constant of Section 4.3. Then synchronize by averaging: replace each temporary matrix by half of itself plus half of the transpose of the other. With $A=W_{temp}^{\ast }\in {\mathbb{R}}^{d_h\times d}$ and $B=W_{temp}\in {\mathbb{R}}^{d\times d_h}$:

$$W_{next}^{\ast }=\frac{1}{2}\left(A+B^{\top }\right),W_{next}=\frac{1}{2}\left(A^{\top }+B\right)$$

Why does this repair the constraint instead of just softening the damage? Check the transpose of the second expression:

$${\left(W_{next}\right)}^{\top }=\frac{1}{2}\left(A+B^{\top }\right)=W_{next}^{\ast }$$

Transposing swaps $A^{\top }$ back to $A$ and $B$ back to $B^{\top }$, which returns exactly the first expression. So after synchronization, $W_{next}^{\ast }$ and $W_{next}^{\top }$ agree identically and exactly, by construction, no matter what the two gradients did. The class phrasing "both become half of each temporary update" is this averaging rule; writing the two results as one line is shorthand for this pair, since $W_{next}^{\ast }$ lives in ${\mathbb{R}}^{12\times 4}$ and $W_{next}$ in ${\mathbb{R}}^{4\times 12}$ — they are transposes, not the same array. Shape check: if $W_{temp}^{\ast }\in {\mathbb{R}}^{12\times 4}$ and $W_{temp}\in {\mathbb{R}}^{4\times 12}$, then $W_{temp}^{\top }\in {\mathbb{R}}^{12\times 4}$, so the sum and average live in ${\mathbb{R}}^{12\times 4}$, matching $W^{\ast }$.

4.4.3 Student Question: What Happens to Training Dynamics?

4.4.4 Capacity, Cross-Validation, and Honest Expectations

Tying cuts the number of free parameters roughly in half. Capacity, here, means how many weights the model owns; fewer weights mean less capacity, and less capacity usually generalizes better. Count for the class example: untied, the encoder owns $4\times 12$ weights and the decoder another $12\times 4$, giving $96$. Tied, both halves share one matrix: $48$. Half the knobs, same wiring pattern.

This is also the right moment to recall how to judge any autoencoder fairly: use cross-validation. Training optimizes reconstruction error on the subset of data used for training, but the honest number comes from a held-out dataset the network never saw. Typically the held-out reconstruction error exceeds the training reconstruction error, and tying improves exactly that held-out number.

And a dose of statistical honesty: phrases like "possibly, most likely better" are the precise level of promise a machine learning method can make. This remains a statistics-driven technique rather than deterministic engineering; you cannot forecast exact performance in advance.

4.5 Contractive Autoencoders

The idea: the encoded representation should not change unless the input changes significantly. Take a small example with three encoding nodes $h_1,h_2,h_3$ fed by inputs $x_1,x_2,\dots ,x_N$. For every hidden unit and every input coordinate, form the partial derivative $\partial h_j/\partial x_i$, square it, and sum over all inputs and all hidden units. Stated in words from class: take this partial derivative of $H$ with respect to $X$, square it, and sum over all inputs and hidden nodes; this has got to be minimized.

4.5.1 The Jacobian Penalty

Those partials arrange into a matrix — the Jacobian of the hidden outputs with respect to the input. The Jacobian is a grid of slopes: row $j$, column $i$ answers "if input coordinate $x_i$ moves by a tiny amount, how much does hidden output $h_j$ move?"

$${\mathcal{J}}_H(x)=\frac{\partial h}{\partial x}\in {\mathbb{R}}^{d_h\times d_x},{\mathcal{L}}_{contractive}=\lambda \sum _{j=1}^{d_h}\sum _{i=1}^{d_x}{\left(\frac{\partial h_j(x)}{\partial x_i}\right)}^2$$

where $d_h$ counts hidden units, $d_x$ counts input dimensions, and $\lambda$ again balances the penalty against reconstruction error. Minimizing the sum of squares of every Jacobian entry forces those sensitivities toward zero: the hidden values stop responding to wiggles in the inputs.

For the single-layer encoder used throughout this lecture, $h=g(W^{\top }x+b)$, the entries have a closed form worth seeing once. The weighted input to unit $j$ is $a_j={\sum }_{k=1}^{d_x}{W}_{kj}{x}_k+b_j$, so by the chain rule:

$$\frac{\partial h_j}{\partial x_i}=g^{\prime }(a_j)\frac{\partial a_j}{\partial x_i}=g^{\prime }(a_j)W_{ij}$$

and squaring and summing collects into column norms:

$$\sum _{j=1}^{d_h}\sum _{i=1}^{d_x}{\left(g^{\prime }(a_j)W_{ij}\right)}^2=\sum _{j=1}^{d_h}{g}^{\prime }(a_j{)}^2\left(\sum _{i=1}^{d_x}{W}_{ij}^2\right)=\sum _{j=1}^{d_h}{g}^{\prime }(a_j{)}^2\Vert W_{:j}{\Vert }_2^2$$

Read this as a product of two brakes per hidden unit: $g^{\prime }(a_j{)}^2$, how awake the unit's activation currently is, times $\Vert W_{:j}{\Vert }_2^2$, the total squared strength of its incoming weights. The contractive penalty pushes down whichever brake is cheaper — calm the activations or slim the weights. One subtlety follows immediately: an activation sitting in its saturated flat region has $g^{\prime }\approx 0$ and contributes nothing, which is why the penalty must be balanced against reconstruction error rather than minimized alone.

4.5.2 A Quantized View of the Code

What does the trained representation look like? The encoding behaves like a high-dimensional quantized version of the original input space: nearby inputs collapse toward shared codes, the way quantization snaps nearby values onto one level. Decoding then acts as dequantization — the decoder maps those flattened hidden outputs back to reconstructions that stay close to the original data. Training a contractive autoencoder means creating exactly this: a quantized-looking hidden representation whose decode still reconstructs faithfully.

An everyday picture: rounding temperatures to the nearest degree. Readings of 20.4°, 20.6°, and 20.5° all snap onto 21°... or onto 20°, depending where the boundary sits; small measurement jitter vanishes because neighbors share a label. Decoding is then like reporting the typical weather associated with that rounded value. The analogy breaks in one honest way: the network's rounding boundaries are learned curved surfaces, not fixed grid lines, and they bend wherever the data demands finer distinctions.

4.5.3 When the Penalty Gets Too Strong

Watch the strength knob $\lambda$. Moderate $\lambda$ buys insensitivity: the code ignores nuisance variation while still separating genuinely different inputs. Push enormous importance onto this penalty and everything freezes.

Real-world placement: learned codes that ignore small perturbations matter wherever embeddings feed a downstream system — face-recognition pipelines, for instance, need identity codes that barely move when lighting shifts by a shade or a camera sensor jitters, which is precisely the insensitivity this penalty trains.

4.6 Denoising Autoencoders

A denoising autoencoder (DAE) encodes data so that unwanted noise variations get suppressed in the reconstruction. The twist sits in the training pipeline. Start with original unlabeled data. Corrupt it deliberately with noise. Encode the noisy version. Decode it. But define the loss so the reconstruction must match the original clean data — not the noisy input that actually entered the encoder.

4.6.1 Corrupt, Encode, Decode, Compare with the Clean Original

Keep the target straight and everything else follows: the noisy version plays the role of input only; the clean original plays the role of label. The reconstruction error is measured against what was never seen by the encoder. This is why a DAE needs no human labels — the clean image is its own teacher — yet the setup behaves like supervised learning, with corruption playing the role of data collection.

A one-line mental model: an ordinary autoencoder asks "can you repeat what I said?"; a denoising autoencoder asks "can you say what I meant, even though I mumbled?" The second task cannot be solved by copying, so the code has to carry meaning rather than raw values.

4.6.2 Gaussian Noise and Salt-and-Pepper Noise

Two corruption styles came up, and the choice matters.

Gaussian noise. Unroll an image into one long vector of pixel values. Call the random generator in the scikit-learn toolbox and add uncorrelated Gaussian noise to each feature element independently:

$${\tilde{x}}_i=x_i+{ϵ}_i,{ϵ}_i\sim \mathcal{N}(0,{\sigma }^2)\text{ per feature}$$

with zero mean and standard deviation $\sigma$ on each element, where $\sigma$ is chosen by the designer. Every feature dimension gets corrupted — no pixel escapes. Values can drift outside the valid pixel range, so in practice you clip back to the legal interval after adding noise.

Salt-and-pepper noise. The name is literal: pepper stands for black, the value 0; salt stands for white, the value 255. The label "color noise" was avoided on purpose — salt-and-pepper is the standard name. Formally, each pixel $x_i$ of the corrupted copy $\tilde{x}$ obeys:

$${\tilde{x}}_i=\{\begin{array}{ll}{x}_i,& \text{with probability }1-q\\ 0\text{ or }255,& \text{with probability }q\end{array}$$

where $q$ is the corruption probability per pixel, set by the designer. Most of the image survives unchanged, with sprinkles of salt and pepper sprayed across it.

4.6.3 Worked Application: Restoring Old Family Photographs

Why train on synthetically damaged photos? Here is the classic scenario from class, walked end to end.

4.6.4 Student Questions and Answers

Several students circled the same worry from another side — what the zeroed pixels are for:

When to pick which? If your deployed inputs arrive already noisy, train a DAE with that same noise; if they are clean but you fear sensitivity to tiny perturbations, contract the Jacobian.

4.6.5 Reading the Learned Filters on Handwritten Digits

One displayed example trains a denoising setup on MNIST handwritten digits — numbers 0 through 9 as 28 by 28 images, pixel values 0 or 255, possibly binarized further to plain 0/1. In a supervised-flavored variant, the decoder output layer carries one node per class: for an input digit 3, the corresponding node should activate while others stay quiet — the decoder effectively indicates which class the input belongs to, so this autoencoder trains in a supervised manner. Adding Gaussian distributed noise damages every pixel — zero mean, some standard deviation — yet the denoiser still trains against the clean originals.

After training, visualize the learned weights. Picture each filter drawn back on the 28 by 28 grid: dark and bright patches arranged along short diagonal and curved segments, resembling pen strokes — little edge detectors matching the strokes that draw handwritten characters, edges like those familiar from computer vision. Run the same setup with L2 weight decay instead, and the filters take on a granular, speckled texture, because weight decay spreads error reduction across all weights rather than sharpening a few. Either way, the hidden neurons behave like edge detectors, and those edges correspond to pen strokes.

Real-world placement: MNIST-style denoising remains the classic demonstration that unsupervised autoencoders discover edge-like visual features without a single label — the same edge vocabulary later exploited by convolutional networks. Beyond teaching examples, the corrupt-clean training recipe directly powers photo-restoration tools that strip grain and speckle from scanned prints.

4.7 Sparse Autoencoders

The sparse autoencoder keeps the hidden layer highly overcomplete — many more coding units than inputs — and relies on a sparsity penalty to block trivial copying. Without any regularizer, an overcomplete autoencoder reproduces the identity: the reconstruction ${\bar{x}}_i$ equals $x_i$ for every feature, since every value can flow straight through. Sparsity changes the incentive: each hidden node may fire — output a high value near one — only for a small fraction of training signals.

Once the network internalizes this budget, every encoding comes out mostly zeros with ones sprinkled at a few positions — and the positions arrange themselves so that decoding still recovers the original data to a large extent.

4.7.1 Firing Rates and the Sparsity Index

Everything hangs on measuring how often a unit fires. Let $m$ be the number of training points and let $h_L(x^{(i)})$ denote the output of hidden unit $L$ when input $x^{(i)}$ passes through. Average that hidden activation over all patterns:

$${\hat{\rho }}_L=\frac{1}{m}\sum _{i=1}^m{h}_L(x^{(i)})$$

Stated in words: sum this L-th hidden activation over all patterns, divide by $m$, and you get a number between 0 and 1 indicating the fraction of training points for which this hidden output fires. This ${\hat{\rho }}_L$ ("rho-hat") is an empirical mean — measured from the activations actually flowing through the network. Against it stands a designer-chosen target $\rho$ (plain "rho"), supplied externally: you decide the sparsity level your architecture should enforce, typically something small like 0.1 or 0.2, possibly smaller when the training data is large. The constraint applies to every hidden node, not just one — there is one measured rate and one target per unit.

4.7.2 The KL Divergence Penalty

The penalty pushing each empirical firing rate toward its target is the KL divergence, borrowed from previous coursework. Summed over all $K$ hidden units ($K$ is the total number of hidden nodes):

$${\mathcal{L}}_{sparse}=\sum _{L=1}^K\left[\rho \log \frac{\rho }{{\hat{\rho }}_L}+(1-\rho )\log \frac{1-\rho }{1-{\hat{\rho }}_L}\right]$$

A note on the logarithm's base, since class weighed the options openly: the choice does not change the mathematics. Changing the base multiplies every term by the fixed constant $1/\ln (\text{base})$, which is indistinguishable from rescaling $\lambda$; the location of the minimum never moves. Standard references define the KL divergence with the natural logarithm, so treat $\log$ here as natural log and move on.

Each term compares two distributions over "fire or stay silent": the target distribution $(\rho ,1-\rho )$ provided by the designer, and the measured distribution $({\hat{\rho }}_L,1-{\hat{\rho }}_L)$ computed by the network. The whole expression reaches its minimum value of zero exactly when ${\hat{\rho }}_L=\rho$ — two probability distributions agreeing perfectly — and it is strictly positive otherwise.

Picture the penalty curve: fix the designer's value at $\rho =0.2$; put the measured rate ${\hat{\rho }}_L$ on the horizontal axis running from 0 to 1, and the penalty on the vertical axis. The curve dips to its minimum of zero right around ${\hat{\rho }}_L=0.2$, forming a valley that rises steeply toward both walls as the measured rate approaches 0 or 1. Any drift to either side costs penalty. Plug in one point to feel it: at ${\hat{\rho }}_L=0.1$,

$$0.2\ln \frac{0.2}{0.1}+0.8\ln \frac{0.8}{0.9}=0.2(0.693)-0.8(0.118)\approx 0.139-0.094=0.044>0$$

That valley shape is why this term works as a sparseness regularizer alongside the standard reconstruction loss.

4.7.3 Gradients and Weight Updates

Weight updates remain straightforward backpropagation on the combined objective — reconstruction loss plus the KL sum. One algebraic convenience helps by hand: expand each logarithm ratio as $\log \frac{a}{b}=\log a-\log b$, splitting every KL term into pieces ready for differentiation.

Differentiating a KL term with respect to the weights uses the chain rule, and the inner derivative with respect to the measured rate comes out as follows, step by step. Write the term in expanded form, then differentiate piece by piece:

$$\begin{array}{rl}\frac{\partial }{\partial {\hat{\rho }}_L}\left[\rho \log \frac{\rho }{{\hat{\rho }}_L}+(1-\rho )\log \frac{1-\rho }{1-{\hat{\rho }}_L}\right]& =\frac{\partial }{\partial {\hat{\rho }}_L}[\underset{\text{constant}}{\underbrace{\rho \log \rho }}-\rho \log {\hat{\rho }}_L\\ & +\underset{\text{constant}}{\underbrace{(1-\rho )\log (1-\rho )}}-(1-\rho )\log (1-{\hat{\rho }}_L)]\\ & =-\frac{\rho }{{\hat{\rho }}_L}+\frac{1-\rho }{1-{\hat{\rho }}_L}\end{array}$$

The two constant pieces vanish under the derivative; differentiating $-\rho \log {\hat{\rho }}_L$ leaves $-\rho /{\hat{\rho }}_L$, and differentiating $-(1-\rho )\log (1-{\hat{\rho }}_L)$ leaves $+(1-\rho )/(1-{\hat{\rho }}_L)$ because the chain rule contributes a factor of $-1$ from $1-{\hat{\rho }}_L$. Sanity-check the sign: when the measured rate sits below target (${\hat{\rho }}_L<\rho$), the negative first term dominates, so raising ${\hat{\rho }}_L$ lowers the penalty — exactly what the valley picture promised.

Then multiply by how the hidden output depends on the weights. The hidden activation takes the familiar affine form:

$$h_L=g\left(W^{\top }x+b\right)$$

where $g$ is the chosen hidden activation function and $W^{\top }x+b$ is the weighted input plus bias. Writing $a_L=W^{\top }x+b$ for the weighted input of unit $L$, the chain rule carries everything through to the parameters:

$$\frac{\partial {\hat{\rho }}_L}{\partial \theta }=\frac{1}{m}\sum _{i=1}^m{g}^{\prime }\left(a_L(x^{(i)})\right)\frac{\partial a_L(x^{(i)})}{\partial \theta }$$

so each weight update combines three factors: how far the unit's firing rate sits from target (the KL derivative above), how awake the activation currently is ($g^{\prime }$), and how much that weight influenced the weighted input. Practice this derivation by hand with pen and paper until it feels routine.

One clarification about balancing terms: wherever an L1 or L2 regularizer appears there is a constant $\lambda$ controlling the trade-off between reconstruction quality and sparseness amount. That same constant applies here too; it was omitted from the displayed formulas for simplicity. If no $\lambda$ is written, assume $\lambda =1$.

4.7.4 Student Questions and Answers

Two questions reached the same confusion — whether adding the $\rho$ machinery means regularizing the divergence all over again:

Real-world placement: sparse codes give complex real-valued data a discrete disguise — most entries zero, a few ones — so clustering original data into a few groups becomes easy, and the scheme supports dictionary learning, where a learned dictionary of prototype atoms combines to rebuild real signals. Sparse encodings also reveal which feature dimensions matter most across the dataset, since only a handful are ever recruited.

4.7.5 A Practice Routine You Can Run Today

Ask ChatGPT to create a tiny sparse network for practice: two inputs, two hidden nodes, and two outputs, with a sparseness target of 0.2 plus a small training set. Then request step-by-step calculation: how the forward outputs are computed, how the loss function is evaluated, how sparseness is measured, and how the weights change iteration by iteration while the overall loss falls. Working through one full cycle by hand cements the whole pipeline.

4.8 Deep Autoencoders

Deep means depth: several layers stacked between input and code and back out, instead of the one encoder layer used in every illustration so far. Single-layer pictures existed only to keep diagrams simple. The advantages of adding layers mirror what supervised deep learning already taught: deeper representations capture structure shallow stacks cannot — low-level pieces compose into parts, parts into wholes, and each level of composition costs only one extra layer of width.

4.8.1 Linear, Shallow, Deep: The Performance Ladder

Three rungs form the comparison ladder:

• At the bottom sit linear autoencoders: both halves are plain matrix multiplications with no nonlinearity. These are, in effect, principal component analysis (PCA) — a linear encoder compresses exactly the way PCA does, projecting data onto its top directions of variance. This is worth pausing on: with linear maps, minimizing squared reconstruction error has the same optimum as PCA's eigen-decomposition, so "autoencoder" and "PCA" name the same machine at the bottom rung.
• One rung up sit shallow nonlinear autoencoders: few layers but a genuinely nonlinear hidden activation $g$, which could be as simple as ReLU, or ELU (exponential linear unit) and relatives met in deep neural network coursework. Nonlinearity lets the code bend the compression surface to fit curved data manifolds, something no straight projection can do.
• At the top, deep autoencoders add hidden layers and perform even better, because each added layer gives the code another chance to re-describe what it sees at a higher level of abstraction.

So the claimed ordering: a deep autoencoder beats the corresponding shallow autoencoder, which beats a linear one. Assignment one asks you to verify this yourself on datasets in your own implementation — the claim becomes your experiment, not just a slide bullet.

Picture the evidence as a bar chart: three bars left to right labeled Linear, Shallow nonlinear, Deep; the vertical axis is reconstruction error on held-out data. Each bar steps down from the one before it, the deepest bar shortest — same code size everywhere, so the only thing changing is representational power.

Node accounting makes the point concrete. In one drawn example, the deep stack contains eleven hidden units in total between input and output layers. To match the same reconstruction error with just one hidden layer, you would need a very large number of hidden units — and then regularization too, otherwise the layer tips into overcomplete territory where copying takes over (Section 4.2's trap all over again). Depth buys expressive reach per parameter; width alone cannot.

When to pick which? If your data is close to flat (linear correlations dominate), PCA is honest and cheap; for images, speech, or any curved manifold, nonlinear depth wins at the same budget.

4.8.2 What Deep Codes Enable

The primary objective stays dimensionality reduction, also called representation learning. On top of that foundation, deep codes support classification, denoising (feed a corrupted input through encoder and decoder, Section 4.6), image inpainting — editing or completing an image — and segmentation performed on deep features.

4.9 Deep Convolutional Autoencoders

For images, replace fully-connected layers with convolutional ones. The resulting architecture partly resembles the CNN designs you already know: blue blocks of convolution followed by batch normalization and ReLU, green pooling blocks that shrink the maps, dropout sprinkled in for regularization, and max pooling stepping sizes down. The encoder is a familiar image-classifier trunk; the twist is what replaces its classification head — a mirrored decoder that grows the picture back.

The whole network is a procedure applied to an image, so walk it as one:

Purpose. Compress a large color image into a small feature map that keeps enough content to rebuild the original, then rebuild it — learning every filter from reconstruction error alone.

Inputs and outputs. In: one color image of roughly $240\times 256$ pixels (three channels). Out: a reconstructed image of the same size, plus the small bottleneck map in between.

Steps. Encoder: alternate convolution (with batch normalization and ReLU) and stride-2 pooling until the map is small; decoder: alternate transpose-convolution upsampling with convolution until the original size returns; compare output to input; backpropagate; adjust all filters. No labels anywhere.

4.9.1 The Encoder Path: Convolution, Batch Normalization, Pooling

Follow the numbers on the color-image example. Each stride-2 pooling halves both side lengths; after four such reductions the side lengths shrink by a factor of $2^4=16$. That small map is the encoded representation produced by the encoder path — the code, in image form.

4.9.2 Batch Normalization Refresher

Because batch normalization appears in every block, refresh what it is and why it exists — this could be a basic question in any machine learning job interview, so it is not a fringe concept.

Mechanically you compute the mean of the mini-batch, subtract it, then rescale so the spread settles near unit variance; two learned parameters per channel (a scale and a shift) let the network undo the normalization where it helps. Batch normalization also carries non-trainable running statistics, accumulated during training and used at inference. It is close to a must for mini-batch-based training of deep networks, letting models converge within noticeably fewer epochs.

A related history lesson on generalization methods, since several students asked:

4.9.3 The Decoder Path: Transpose Convolution Upsampling

The decoder reverses the journey, and here sits the genuinely new piece: instead of striding down, stride up. Where the encoder pooled from large image to small feature map, the decoder applies fractionally strided convolution — also called transpose convolution — with upsampling steps of two, mirroring each stride-2 pooling. Intuition for the operation itself: ordinary convolution asks "given a patch, how strongly does it match my filter?" and produces one number; transpose convolution runs the same wiring backwards, asking "given one number, paint the patch it should have come from" — each input value spreads a learned pattern outward, overlapping neighbors stitch together, and the map grows. The filters doing the painting are learned, like everything else here.

The map grows back along exactly the mirror of the encoder's staircase: about $15\times 16$ becomes $30\times 32$, then $60\times 64$, then $120\times 128$, continuing until the original $240\times 256$ color image returns. Transpose convolution was taught in the deep neural network course even though none of its architectures used it — now you see where it earns its place. At the end, compare the reconstructed image with the original and train by minimizing reconstruction error, adjusting the filters in the encoding layers as usual. No labels are needed anywhere in the whole structure.

Real-world placement: this encoder-decoder shape with a transpose-convolution decoder powers modern image inpainting tools — networks that edit photographs by filling damaged or removed regions with plausible content — because growing a clean image out of a compressed code is exactly what the decoder practices all day.

4.9.4 Comparing Reconstructions: Deep Autoencoder versus PCA

One displayed comparison uses MNIST handwritten characters at 28 by 28 pixels. Three reconstructions appear side by side, all squeezed through a 30-dimensional code: a 30-dimensional deep autoencoder, 30-dimensional logistic PCA, and 30-dimensional plain PCA. The deep autoencoder reconstruction is visibly sharper and clearer than either PCA variant — strokes stay crisp while the linear versions blur toward average-looking digits. Push further to deep convolutional networks and performance improves again. The takeaway stands for your own assignment experiments: nonlinear codes beat linear ones at equal code size, and convolutional nonlinear codes beat dense ones on images.

4.10 Preview: Variational Autoencoders

The family is the variational autoencoder (VAE), which opens right after the mid-semester examination. It carries applications into generative modeling and counts among the classic generative AI designs. The VAE still belongs to the autoencoder category — encoder, code, decoder, reconstruction error — and the difference is entirely in the regularizing functional, which takes a different type.

4.10.1 A Penalty That Shapes a Distribution

Every regularizer met today pushed on values: weight sizes (L1/L2), tied structure, derivative magnitudes (contractive), firing rates (sparse), or noise robustness (denoising). The VAE constraint targets the latent space itself: the probability distribution associated with the auto-encoded space is forced toward a zero-mean Gaussian. Train with that functional and you get a compact encoder whose latent space supports something no ordinary autoencoder offers — sampling.

Why does distribution-shape matter so much? An ordinary autoencoder's codes scatter wherever they please: between two occupied code regions lie empty gaps, and if you invent a code point inside a gap, the decoder produces garbage — nobody ever trained it there. A Gaussian-shaped latent space has no gaps: every point sits near training codes, so every point you can name decodes to something sensible.

A map analogy helps: an ordinary autoencoder builds cities — dense blocks of valid codes separated by wilderness where decoding fails. A variational autoencoder builds road cover across the whole country; drive (sample) anywhere and you still find a usable destination. The analogy breaks where all analogies about learned spaces do: the "roads" are high-dimensional curved regions, not flat lines on paper.

4.10.2 Sampling New Faces from the Latent Space

Here is the classic demonstration. Train a variational autoencoder on a large set of human faces varied along many axes: male and female, some wearing sunglasses and some not, some smiling and some not, different skin tones, some looking straight at the camera and some sideways. After training, draw a random sample $z$ from the latent space — that is, pick coordinates from a standard zero-mean Gaussian — and pass $z$ through the decoder.

The output might be a smiling woman wearing sunglasses — a combination absent from every training image, yet completely meaningful. The model recombined learned factors of variation (smile from some images, sunglasses from others) into one coherent new arrangement.

That is the essence of a generative model: manufacture new data that is valid and useful yet never appeared in the training set, created purely by sampling the learned latent space. So the VAE provides a powerful paradigm for learning the probability distribution of data — the continuation of the autoencoder story with a distribution-shaped penalty.

Visual intuition for the mechanism: imagine the latent plane as a grid over which decoded outputs morph smoothly — walking east turns neutral expressions into smiles, walking north adds sunglasses. Landmarks sit at training clusters; the paths between them are what sampling travels. One-sentence takeaway: shaping the latent distribution into a smooth blob turns a compressor into an inventor.

4.11 Autoregressive Models and Likelihood Thinking

The first family of models for this job is autoregressive and likelihood-based. Likelihood-based thinking goes back to earlier statistics coursework; here it powers learning probability distributions of data directly.

4.11.1 What Autoregressive Means

The name says it: a particular variable depends on whatever variables appeared in the past — never on the future. Each prediction reaches backward only. In symbols, a model of the sequence $x_1,x_2,\dots ,x_n$ is built from conditionals of the form $p(x_i\mid x_1,\dots ,x_{i-1})$: the next value given everything before it. The word "auto" marks that the model's own previous outputs become its future inputs at generation time.

4.11.2 The Familiar RNN Connection

You already know one autoregressive system: the recurrent neural network (RNN) doing next-word prediction. Once trained, feed it a word or a letter; conditioned on that history, it generates the next word, then the next, each output conditioning on everything generated so far, until a stop symbol arrives. Whatever appears at each step depends only on what happened in earlier steps — the autoregressive recipe in action. Text generators you use daily run exactly this loop under the hood.

4.11.3 The Model Roadmap: Masking, WaveNet, PixelCNN

Several specific models fill the coming sessions:

• Masking-based methods. Mask out information from the future so training uses past data only. A mask is just a pattern of blocked connections that makes "no peeking ahead" structural rather than aspirational.
• MADE (Masked Autoencoder for Density Estimation). A classic method: an ordinary feedforward network whose weights are masked so each output sees only earlier inputs. Even if it sees little direct use today, it inspired many methods in service right now.
• WaveNet. An autoregressive model suited to speech signals; a variant still drives text-to-speech products. Real-world placement: the underlying technology supports dubbing — replacing the dialogue track of one speaker with another speaker's voice while keeping the video intact.
• PixelCNN. Convolutional networks performing density estimation on image pixels, introduced next.

4.11.4 PixelCNN's Product Rule over Pixels

How can a CNN estimate a distribution over images? Take $n\times n$ images with pixels $x_1,x_2,\dots ,x_{n^2}$, drawn from many training examples — say 10,000 images. Look at how the intensity of the very first pixel varies across those images and fit a distribution to it. Then predict the second pixel's distribution based on what came before it, and continue pixel by pixel, each one conditioned on its predecessors. Stated in words: the overall distribution of the whole image is the product of these per-pixel distributions, and maximizing that product — the likelihood — uncovers the parameters of the distribution.

This is the chain rule of probability applied along pixel order:

$$p(x)=\prod _{i=1}^{n^2}p\left(x_i\mid x_1,\dots ,x_{i-1}\right)$$

Two things make this more than bookkeeping. First, the factorization is an exact identity — true for every joint distribution, with no independence assumptions anywhere. All modeling freedom hides inside the chosen form of each conditional, not in the product itself. Second, training turns multiplication into addition: maximize $\log p(x)$, equivalently minimize the negative log-likelihood $-{\sum }_{i=1}^{n^2}\log p(x_i\mid x_1,\dots ,x_{i-1})$, which decomposes into per-pixel terms a CNN can compute in one pass when its masks hide the future. The convention fixing "the past" is a raster scan: left to right, top to bottom, as reading order. Maximizing this likelihood-based objective is the basic principle behind CNN-driven density estimation of images.

4.12 Three Jobs for a Learned Distribution

Three goals justify learning the distribution of your data:

• Synthesizing new data — images, videos, speech, text — where whatever you generate should be a valid sample drawn from the learned distribution.
• Compressing data, because storing the parameters of a learned distribution costs far fewer numbers than storing the raw dataset — that is what constructing efficient codes means.
• Anomaly detection.

4.12.1 Worked Example: Is 25 an Outlier?

Anomaly detection falls straight out of the framework. Suppose these samples arrive from a one-dimensional Gaussian distribution:

$$13,4,7,6,5,12$$

Question: is 25 an outlier? Walk the full path, five steps.

4.12.2 Classroom Exchange: Nearest Neighbors versus Fitted Probabilities

A natural instinct proposed in class was to hunt for anomalies by proximity:

4.12.3 Scaling Up: From Six Numbers to Images

The toy example used six scalar samples. Even a small 28 by 28 image makes $x$ live in 784 dimensions, and a modest 128 by 128 color image with 3 channels pushes dimensionality past 49,000. Two hard requirements follow. First, the method must work well in high-dimensional space even when the training set covers that space sparsely — learn the distribution from sparse high-dimensional data and still expect good generalization. Second, learning must be computationally efficient, and the chosen form of distribution must be easy to sample from.

4.12.4 Sampling Speed Matters Too

Ease of sampling is not enough; speed counts as well. Generating new data should happen fast — ideally all pixels produced in parallel rather than one after another. This requirement quietly rules some families in and others out: autoregressive models generate sequentially by construction, which is exactly the behavior Section 4.13 will return to.

4.13 The Wishlist for Modern Generative Models

Assemble every requirement raised across the session and you get the checklist against which generative models are judged.

4.13.1 Six Desirable Properties

Read the table as a set of simultaneous constraints: each row alone is achievable today; all six rows together have never been met by one system.

4.13.2 Why No Single Model Wins

No existing model satisfies every item fully — which explains why the landscape keeps multiplying: ChatGPT, Anthropic, Gemini, and many others coexist, each strong somewhere on the list and weaker elsewhere.

Watch ChatGPT generate a complex design document: the whole document never appears in one shot; it emerges iteratively, sentence after sentence. That iterative behavior is the slow sequential sampling showing through — the same autoregressive, past-only generation met in Section 4.11. These systems still do remarkable work, but many dimensions remain open for improvement — and that gap is exactly where research continues.

The practical reading of this section: when you meet any generative model in the coming sessions, interrogate it against these six properties. Asking "where does it sit on the checklist?" replaces vague impressions with a precise comparison, and it predicts exactly where the next generation of systems will try to improve.

Real-world placement: the checklist explains product strategy across the AI industry — voice assistants prize sampling speed, image tools prize sampling quality, and API providers prize training efficiency, so each vendor's flagship reflects a different bet about which property users will pay for.

Exam Guidance Summary

• Backpropagation mastery is assumed in the exam: expect to compute weight updates for autoencoder architectures and for any architecture covered in this course. Review it until applying it is mechanical.
• The mid-semester examination falls roughly four to five weeks after this session; everything from the regularized-autoencoder stretch onward is fair game.
• Assignment one carries two verification tasks: confirm empirically that deep autoencoders beat shallow nonlinear ones, which beat linear (PCA-style) ones; and train a deep convolutional autoencoder to produce high-quality reconstructions, with design details intentionally left open for experimentation.
• Past question papers were promised for upload as practice material; problem-solving webinars follow later, including sessions devoted to working through past papers.
• ChatGPT is endorsed for generating practice problems in this area — its solutions are mostly correct — but always check that its method matches the class method, since multiple valid methods can give slightly different results.
• Batch normalization is named explicitly as a basic machine learning interview question, not a fringe concept; know its purpose (stable per-layer mini-batch statistics), where it applies (per mini-batch during training), and its effect on training speed (fewer epochs) without needing formulas.

Key Industry Applications

• Text-to-speech and dubbing. WaveNet-style autoregressive models still drive speech generation products, including dubbing pipelines that replace one speaker's dialogue with another speaker's voice while keeping the video untouched.
• Photo restoration. Denoising autoencoders trained on synthetically corrupted fresh photographs can clean digitized old family prints whose damage resembles salt-and-pepper noise — the corrupt-clean training recipe at work.
• Image inpainting. Deep convolutional encoder-decoder pairs underpin tools that edit or complete images, growing clean content out of a compressed code via transpose convolution.
• Practice tooling. ChatGPT serves as an on-demand problem generator for this subject area, with method verification left to the student.
• The generative model industry. No single system meets every desirable property — accurate modeling, fast training, expressiveness, sampling quality, sampling speed, compression — which is why ChatGPT, Anthropic, Gemini, and many competing systems coexist, each strong on different axes of the checklist.