Variational Autoencoders

9.1 Autoencoder Recap and the Generative Model Landscape

9.1.1 What a Vanilla Autoencoder Does

Here is the opening puzzle of this lecture: a network trained only to copy its input seems doomed to merely echo the past — so how could anything built that way invent new data? The answer starts with what the copying actually forces the network to learn.

An autoencoder is a network that squeezes data into a smaller code and then rebuilds the data from that code. Think of retelling a two-hour film in one minute: you cannot keep every scene, so you must decide which details carry the story. The autoencoder is forced into the same kind of decision, but it must learn the decision rule by itself. Say the original data lives in $D$ dimensions — $D$ counts the number of descriptive numbers per item, such as pixel intensities or measured features. The network transforms each item into a reduced representation of dimension $d$, where $d$ is much smaller than $D$. Training shapes this transformation so that, starting from the $d$-dimensional code, you can recreate data close to the original in $D$ dimensions. The closeness is measured over the entire training set, not just one point — a model that copies one item perfectly while failing on the rest has not learned anything useful.

Two parts do the work:

• The encoder maps the input to a latent code. The word latent means hidden: the code appears nowhere in the raw training set. It is a learned representation, a transformation of the original data. If the encoder is a function $f_{\varphi }$ with parameters $\varphi$, then for input $x_i\in {\mathbb{R}}^D$ the code is $z_i=f_{\varphi }(x_i)\in {\mathbb{R}}^d$.
• The decoder passes that transformed data back out, aiming to reconstruct the input. Writing the decoder as $g_{\theta }$ with parameters $\theta$, the reconstruction is ${\hat{x}}_i=g_{\theta }(z_i)\in {\mathbb{R}}^D$.

We train the encoder-decoder combination by minimizing a loss function, typically the gap between reconstructed data and original data.

Real-valued data can take any continuous number from minus infinity to plus infinity; nothing restricts it to integers. Pixel intensities, sensor readings, and feature measurements are all like this. For such data the usual choice is the mean squared error between reconstruction and input, averaged over the training set:

$${\mathcal{L}}_{\text{AE}}=\frac{1}{N}\sum _{i=1}^N\Vert x_i-{\hat{x}}_i{\Vert }^2$$

Here $N$ is the number of training points, $x_i\in {\mathbb{R}}^D$ is an original data point, and ${\hat{x}}_i\in {\mathbb{R}}^D$ is its reconstruction. The double bars measure the Euclidean norm: square each coordinate's difference, add them up, and take the square root — squaring first makes every mismatch count positively and punishes big errors harder than small ones. In plain words: square the difference between what we rebuild and what we had, then average. We lower this error by changing the parameters of the decoder and the encoder through standard backpropagation.

One assumption sits quietly inside this choice of loss: mean squared error fits real-valued targets. If the data were categorical — letters, labels, discrete tokens — squared distance would make little sense, and a cross-entropy-style loss would take over. That boundary matters later, when this lecture meets data that refuses to be continuous.

9.1.2 Regularized Variants Already Covered

Earlier material covered several variants built on this skeleton, each adding one extra pressure to training:

• L1/L2-regularized autoencoders add a penalty on the weights themselves, keeping them small so the learned map stays smooth rather than memorizing.
• The sparse autoencoder pushes most code units toward zero so only a few stay active. At any moment the code looks like a nearly empty vector with a handful of live entries — a constraint that often yields interpretable features.
• The denoising autoencoder learns to clean corrupted inputs back into clean ones: feed it a noisy version, ask for the original. Because the corruption changes every time, the network cannot copy mechanically; it must recover structure.

All of these still share one job: map input to a compact code and reconstruct the input from it. And all share one ceiling — none of them can invent new data. Every decoder output is a reply to some input that already exists. Nothing in their training asks them to fill in the space between known items. Closing that gap is the business of generative models.

9.1.3 Three Ways to Generate Data: Autoregressive, Flow, Variational

A variational autoencoder, shortened to VAE, differs from every autoencoder above in capability and features. A VAE can produce data semantically similar to its training set yet different from every training point. It has the ability to generate new data. In that sense it sits alongside two other generative families covered before: autoregressive models and flow models.

Each family makes a different trade-off along the same axes — how fast samples arrive, how honestly they report probabilities, how stable training is, and how large the latent space must be:

Read the table column by column. An autoregressive model generates one element at a time, each new piece conditioned on what came before. Modern text systems work this way: Claude or ChatGPT emit output line by line, even word by word, never in one shot. Their strengths are real — likelihood maximization, exact probabilities for produced data, stable training. Their weakness is speed, because sequential generation is slow: element ten thousand cannot start before element nine thousand nine hundred ninety-nine finishes.

A flow model generates in parallel rather than sequentially, which makes it fast. Two characteristics matter here. First, the quality of flow-model output often trails what autoregressive models reach, though some progressive variants close much of that gap. Second — and this is a structural signature — the latent space of a flow model has the same dimensionality as the data itself. You sample a point from that data-sized latent space and apply an inverse transformation to get the generated item. For a million-pixel image, that means sampling a million-dimensional noise vector: no compression happened anywhere.

The VAE takes a third path. Generation happens at one shot, in parallel; nothing about it is sequential. It too rests on likelihood maximization. It can report the probability of new data, with some approximation. The cost sits in the mathematics. The derivation is crooked and complex, demanding heavy symbol pushing. And the outcome is not exact likelihood maximization but approximate likelihood maximization. In its original form the VAE produces samples below autoregressive quality, yet the samples arrive in parallel. One more family note: variants of hierarchical autoencoders have also been proposed that generate data in parallel while keeping strong visual clarity on images.

When to pick which: if you need the sharpest possible samples and can wait, go autoregressive; if you need fast sampling with exact likelihoods and do not mind a full-sized latent space, go flow-based; if you need fast generation plus a compact, structured code you can analyze and edit, the VAE earns its complexity.

Real-world: ChatGPT and Claude generate text sequentially, one token after another — the autoregressive pattern. Image-oriented generative development draws heavily on two classical techniques, the VAE and the GAN, and much of modern generative AI is rooted in that pairing.

9.2 Loss of Semantics in Standard Latent Spaces

9.2.1 Why an Unstructured Latent Space Fails

Every autoencoding compresses $D$ dimensions into $d$, so information loss is unavoidable — that much is arithmetic. The deeper problem is different. In all the standard autoencoders, the geometry of the latent space is unmanaged: training never tells the encoder where to place codes relative to each other.

A useful picture is a library shelved by accession number instead of by subject. Every book sits somewhere, but two neighbors on the shelf can be a cookbook and a calculus text, because nothing forced similar topics together. A reader who wanders one shelf over finds no promise of anything related. The standard autoencoder's latent space is exactly such a library — codes are placed wherever reconstruction pressure happens to drop them, and adjacency carries no meaning.

Take two points that sit close together in the latent space — close meaning a small L2 distance between them, that is, the straight-line length of the gap you compute by squaring each coordinate difference, adding them, and taking the square root. Once you decode those nearby codes, the outputs may not resemble each other at all. That failure is called loss of semantics: the neighborhood structure of the code space tells you nothing reliable about the data space. Two latent points centimeters apart can decode to a photograph of a dog and a photograph of an aeroplane.

A VAE attacks this directly. During training it does extra work that forces the latent space to become regular, structured, and well behaved. Regularity means precisely this: if two points are close in the latent space, their decoded images stay close in the $D$-dimensional data space. Preserve that property and the code space carries meaning — wandering one shelf over always lands you on a related book.

9.2.2 Worked Illustration: Animals and Vehicles in One and Two Dimensions

Consider a small set of object classes: dog, bird, cat, car, and aeroplane. Each class is represented by some feature vector. Let the original dimension be $D=9$ — count the descriptive features of shape, parts, texture, and so on: nine of them, say number of legs, presence of wings, presence of wheels, fur versus metal, engine sound, and the rest.

d = 1 axis:
  living end  <--- car   plane | bird  cat  dog --->  (positions illustrative)

 vertical axis: alive (top) vs inanimate (bottom)

        alive ↑
   bird ●     │     ● dog/cat
              │
   ───────────┼─────────── → flies (horizontal axis)
              │
   plane ●    │     ● car
       ↓ inanimate

9.2.3 Worked Illustration: Shapes and Colors Off the Training Distribution

Here is the sharpest way to see the failure.

9.2.4 What Regularity and Structure Mean

A variational autoencoder enforces regularity and structure in the latent space so this cannot happen. Sample any point from the latent space a trained VAE builds. Decode it. You get something similar to the training set yet completely different from any single training item. The output is never random; it stays along the same line as the training data. Where the standard autoencoder returned junk for the between-points sample, the VAE returns a meaningful blend — a shape leaning triangle-ish, a color between the neighbors'.

Visualize the contrast as two scatter plots of latent codes. In the standard-autoencoder plot, codes cluster into scattered islands separated by wide empty stretches — decode from an empty stretch and anything can come out. In the VAE plot, the islands have swollen and merged into one smooth cloud centered at the origin, and every point inside it decodes to plausible data. The takeaway sentence for both plots: closeness in code space predicts closeness in data space, both for stored codes and for fresh samples.

Real-world: this distinction decides whether a learned representation supports interpolation-based tools — image morphing, style blending, synthetic-data synthesis all decode off-distribution points, and all silently assume a regular latent space. Recommendation systems that blend user-preference vectors and image editors that slide "age" or "smile" dials are standing on exactly the property this section defined.

9.3 Core Design of the Variational Autoencoder

9.3.1 Learn the Distribution, Not the Code

The last section ended with a demand: force the latent space to decode sensibly everywhere, not just at visited codes. The VAE meets that demand with one structural change so fundamental that everything else follows from it.

In a standard autoencoder you take an image $x$, produce a latent representation $z$, reconstruct $\hat{x}$, and drive $\hat{x}$ toward $x$ by minimizing the squared gap. The VAE changes what the encoder emits. It does not output the latent representation itself. It outputs the parameters of a probability distribution over the latent space. Instead of producing $z$, the network produces $p(z\mid x)$ — the distribution of the latent given the input. Concretely, for each input the encoder names one Gaussian: its center and its spread.

Once training gives you this distribution, you can draw a sample from it. Pass that sampled value through the decoding process and you get the reconstruction. The training target stays familiar: the reconstructed input should sit very close to the original input. So the first distinction is exact — the variational encoder hands you distribution parameters rather than latent values. Where the plain encoder answered "the code is here," the variational encoder answers "the code lives somewhere in this neighborhood, centered here, with this much uncertainty."

9.3.2 The Gaussian Prior and the Two-Part Loss

The second design move preserves semantics. The VAE assumes a prior distribution for the latent variable $z$, and that prior is the normal Gaussian distribution — the bell curve with mean zero and unit variance in every direction. A prior is the distribution we believe $z$ follows before seeing any data. During training, the model works so that the learned encoding distribution $p(z\mid x)$ stays close to this normal Gaussian. Why does that help? Because a code space filled with well-placed Gaussians leaves no wild gaps: sample anywhere reasonable, decode, and land near meaningful data. Every neighborhood of the prior carries some encoder mass, so no decoded sample lands in uncontracted territory.

That sets the shape of the loss. The loss function is designed so that its value drops when two things happen together. One, the reconstruction error becomes small. Two, the latent distribution the encoder produces stays close to the normal Gaussian. Equivalently, the loss penalizes any deviation of the encoding distribution away from the normal Gaussian, alongside the usual reconstruction penalty. The regularization applied in a VAE is exactly a functional tied to the deviation of the latent distribution from that fixed prior.

Written pedagogically, the quantity to minimize combines both parts:

$$L_{\text{VAE}}=\underset{\text{reconstruction error}}{\underbrace{\sum _{i=1}^N\Vert x_i-{\hat{x}}_i{\Vert }^2}}+\underset{\text{regularization}}{\underbrace{D_{\mathrm{KL}}(q_{\varphi }(z\mid x)\Vert p(z))}}$$

The first term sums squared reconstruction gaps over the training set — the same quantity as ${\mathcal{L}}_{\text{AE}}$ from Section 9.1 without the $1/N$ factor. The second term is the Kullback-Leibler divergence, introduced properly in Section 9.4. It measures the gap between the encoder distribution $q_{\varphi }(z\mid x)$ and the assumed prior $p(z)$, taken to be the zero-mean normal Gaussian. Minimizing the whole thing says two things at once: reconstructions sit close to originals, and the encoded latent distribution resembles the prior.

9.3.3 Encoder Outputs and Their Shapes

Look inside the encoding process. Original data enters, and the encoder emits the parameters of the latent distribution, denoted ${\mu }_X$ and ${\Sigma }_X$. Dimension bookkeeping matters. If the input dimension is capital $D$, the latent dimension is small $d$, far smaller. Concretely:

• ${\mu }_X$ is a $d$-dimensional vector — the mean of the latent Gaussian, the center of the cloud of plausible codes for this input.
• ${\Sigma }_X$ is a $d\times d$ covariance matrix for that Gaussian — entry $(j,k)$ records how latent dimensions $j$ and $k$ vary together.
• ${\Sigma }_X$ is treated as diagonal in nature; off-diagonal entries drop out, so each latent dimension keeps its own variance and none co-vary. A diagonal matrix means the Gaussian is axis-aligned: stretching along one latent direction never drags another along.
• The prior against which everything is compared is the standard normal, whose covariance is the identity: ones along the diagonal, zeros everywhere else.

Regularization pushes the learned $\mu$ values toward zero and the variances toward the identity pattern, making the whole latent cloud look like the standard Gaussian.

9.3.4 Sampling a Latent Value

How does a concrete latent value come into existence? Draw a random vector and combine it with the learned parameters:

$$z_i={\mu }_X+\epsilon \cdot {\sigma }_X$$

Here $\epsilon$ is a random vector whose entries are independent draws, and ${\sigma }_X$ collects the square roots of the diagonal variances of ${\Sigma }_X$ — each entry scales the noise on its own latent dimension. The in-class description put the components of $\epsilon$ between zero and one; the standard formulation draws $\epsilon \sim \mathcal{N}(0,I)$, a standard normal noise vector, matching the reference text's reparameterization relation $z^{\ast }=\mu +{\Sigma }^{1/2}\epsilon$. Both tell the same story — add randomness scaled to the learned spread onto the learned center — and from here on $\epsilon$ denotes a standard-normal draw.

Every call to the random number generator returns a fresh $\epsilon$, and multiplying it with the covariance-scale factor and adding the mean yields a new sample $z_i$. Feed $z_i$ to the decoder and out comes $\tilde{x}$, a data item numerically different from every training point yet semantically close to them. In strict matrix terms the same idea reads $z={\mu }_{z\mid x}+{\Sigma }_{z\mid x}\epsilon$, with shapes matched so a $d\times d$ covariance multiplies a $d$-dimensional noise vector.

9.3.5 Worked Example: Generating a New Car Design

Train a VAE on a large collection of car images. Training ends. Now call the random number generator; suppose it returns the vector we labeled ${\epsilon }_{\text{car}}$ above. Form the sample as ${\epsilon }_{\text{car}}$ scaled by ${\Sigma }_X$ and shifted by ${\mu }_X$ — exactly the arithmetic of the previous subsection, giving $z=[0.26,-0.06]$. Push this $z$ through the decoder. The output is an image of a car — maybe — that matches no single car in the training set. It is a new design of car: perhaps the wheelbase of one training car fused with the roofline of another, because those traits live along nearby latent directions. Repeat the call and you harvest more designs, each pass independent. That loop is how the VAE delivers its generative capability. It also explains why the VAE and the GAN became the two dominant classical engines of image generation. Fresh techniques keep arriving on top of that foundation.

Visualize what the trained latent space looks like as a heat map over the plane: brightness marks where the mixture of per-input Gaussians concentrates. After training, the bright region forms one smooth blob centered at the origin — the KL pressure has squeezed all per-input clouds into overlapping position. Any point you pick inside the blob lies within reach of some encoder distribution, so any decode stays meaningful.

Real-world: this sampling loop is production machinery. Car studios prototype concept designs by sweeping latent samples around a reference vehicle's ${\mu }_X$; game asset pipelines generate texture variations the same way; drug-discovery tools sample molecular latents to propose candidate structures no chemist has synthesized.

9.4 Entropy and KL Divergence

9.4.1 Entropy Refresher

The VAE loss leans on one question, asked straight from the floor: how do you quantify the difference between two probability distributions? The VAE wants the latent distribution to mimic a zero-mean normal Gaussian. That assumption keeps the latent space regular and structured, so similar codes decode to similar data. To demand similarity, we must measure dissimilarity. The tool for that is the KL divergence, and the KL divergence grows out of entropy, introduced in the machine learning course.

Entropy measures indefiniteness — how unpredictable a random variable is. A good everyday anchor: guess-the-card. If I hold a deck where one specific card always appears (deterministic), you never have to guess. If any of fifty-two cards is equally likely, guessing is maximally hard. Entropy puts a number on that hardness.

For a discrete random variable $X$ with probability $p(x)$ at each value $x$:

$$H(X)=-\sum _{x}p(x)\log p(x)$$

The sum runs over every value the variable can take. Each term multiplies a probability by the logarithm of itself — so rare events contribute little weight, and impossible events contribute none at all, since their probability is zero. This quantity is the entropy: a measure of how unpredictable the variable is. Change the logarithm base and you change the unit: with log base 2, entropy is measured in bits, the same currency as computer storage.

9.4.2 Entropy at Its Extremes: Worked Computations

Deterministic system. Let the variable take exactly one value: probability 1 there, probability 0 everywhere else. Nothing is indefinite — you know the value all the time. At the certain value the term contributes $1\cdot \log 1=0$; everywhere else $p(x)$ is 0. Either $p(x)$ is 0 or $\log p(x)$ is 0, so every term vanishes and the entropy equals 0. A deterministic system carries no disturbance and no surprise.

Completely random system. Entropy peaks when probability spreads uniformly over the outcomes. The classic case is a fair coin toss: heads with probability 0.5, tails with probability 0.5. Plug into the formula with log base 2:

$$H=-(0.5{\log }_{2}0.5+0.5{\log }_{2}0.5)=-(0.5\times (-1)+0.5\times (-1))=1$$

All values equally likely, entropy equal to 1 bit — the maximum for this setting. As supporting background: spreading probability uniformly over $K$ equally likely outcomes gives ${\log }_{2}K$ bits, so a fair eight-sided die carries 3 bits; the binary coin toss is the special case $K=2$. So the boundaries are settled: minimum value 0 for a deterministic system, maximum value 1 bit for a completely random binary system.

9.4.3 Why the Minus Sign

The takeaway from that exchange, restated once without the symbols: the flip exists because $\log p\le 0$ whenever $0<p\le 1$ — it converts an always-negative sum into an always-positive uncertainty score, so a bigger number really means more unpredictability.

9.4.4 From Entropy Difference to KL Divergence

Why climb from entropy to divergence? Because entropy measures the information content carried by one distribution, and comparing two entropies compares two information contents. If something is deterministic, its entropy is 0 and it carries no information; if something is completely random, it carries maximal information. Given two distributions $P$ and $Q$, each owns an entropy. The gap between their information contents can be written through expressions of the form $-\sum q(x)\log p(x)$, which mix the probabilities of one distribution with the logarithms of the other. From this core idea of information theory, the KL divergence emerges as the working variant.

The KL divergence — called KLD or KL divergence for short — of one distribution relative to another is assembled from exactly those pieces:

$$D_{\mathrm{KL}}(P\Vert Q)=\underset{\text{cost of using }Q\text{ for }P}{\underbrace{-\sum _{x}P(x)\log Q(x)}}-\underset{\text{entropy of }P}{\underbrace{(-\sum _{x}P(x)\log P(x))}}=\sum _{x}P(x)\log \frac{P(x)}{Q(x)}$$

Read the middle expression first: it subtracts what $P$ already costs from what it would cost to describe outcomes drawn from $P$ while pretending they come from $Q$. Combining the two sums over a common denominator gives the rightmost ratio form. In words: weight the logarithm of the probability ratio $P/Q$ by the probability under $P$, then add up those weighted log-ratios over every value of $x$. This matches the reference definition exactly — $D_{\mathrm{KL}}[p(x)\Vert q(x)]=\int p(x)\log \frac{p(x)}{q(x)}dx$ in the continuous case, with the sum replacing the integral for discrete variables.

The reverse direction swaps the roles:

$$D_{\mathrm{KL}}(Q\Vert P)=\sum _{x}Q(x)\log \frac{Q(x)}{P(x)}$$

Now the weighting distribution flips along with the ratio — a different weighted average, and so a different number.

Check the special cases straight from the formula. If for every value of $x$ we have $P(x)=Q(x)$, the ratio is 1, its logarithm is 0, and the KL divergence equals 0. If the distributions are completely disjoint — some value has $P(x)>0$ but $Q(x)=0$ — the ratio blows up and the divergence takes a very large value, infinite in the limiting sense.

9.4.5 Properties: Non-Negative, Asymmetric, Not a Metric

Three properties decide how KL divergence may and may not be used.

First, it is always non-negative. The reference text proves it cleanly: since $-\log y\ge 1-y$ for every positive $y$, substituting $y=Q/P$ gives

$$D_{\mathrm{KL}}(P\Vert Q)=-\sum _{x}P(x)\log \frac{Q(x)}{P(x)}\ge \sum _{x}P(x)(1-\frac{Q(x)}{P(x)})=\sum _{x}P(x)-\sum _{x}Q(x)=1-1=0$$

Identical distributions achieve exactly 0; different distributions give strictly larger values. That monotone behavior is what a similarity score needs.

Second, it is not symmetric. Compare the two displayed equations above: interchanging $P$ and $Q$ changes which distribution weights the logarithm, so the numerical result changes — the worked example just measured 0.74 versus 0.53 bits. Symbolically, $D_{\mathrm{KL}}(Q\Vert P)\ne D_{\mathrm{KL}}(P\Vert Q)$.

Third, symmetry fails, so KL divergence violates the standard definition of distance in a metric space. In a metric space, the distance between $x$ and $y$ must equal the distance between $y$ and $x$. It is a divergence, a directed measure of how one distribution deviates from another — not a distance in the geometric sense. Keep this straight whenever an exam question asks whether KL is a metric: it is not.

9.4.6 The KL Term as the VAE Regularizer

Now place the tool inside the VAE. The quantity to control is the encoder's latent distribution, written $q_{\varphi }(z\mid x)$. Here $x$ is the input, $z$ is the latent, and $\varphi$ collects the parameters of the autoencoder's encoding side. The target is the normal Gaussian prior. The regularizing functional is:

$$D_{\mathrm{KL}}(q_{\varphi }(z\mid x)\Vert \mathcal{N}(0,I))$$

Training chooses and learns the parameters $\varphi$ so this divergence sinks toward zero. Driving it to zero keeps the latent space close to the normal Gaussian. Sample from that space and decode, and you get meaningful, semantically valid new data. That single term is the entire mechanism by which the VAE regularizes its code space.

9.4.7 What the Regularizer Does to the Latent Space

Picture a two-dimensional latent space trained without the KL pressure. The training codes huddle in scattered islands, and much of the plane is empty. Sample a point in one of those empty stretches, far from every island, decode it, and junk comes out. Pick a point inside an island and decoding reproduces something close to genuine training data — but recreating training data is not the goal. The goal is data similar to the training set yet distinct from it.

With the KL regularization forcing the latent distribution toward the normal Gaussian, the islands swell and merge into smooth coverage. On the scatter-plot picture from Section 9.2: the empty stretches fill in until one connected cloud spans the region around the origin. Now interpolation works. Train a VAE on many frog images and many car images — sports cars, sedans, hatchbacks, frogs from varied environments. Sample a latent point between the frog region and the car region, then decode it. Nearer the frog side, you get a frog that looks like a car. Nearer the car side, you get a car that looks like a frog. No other framework in this course offers that interpolation capability. The same story plays out with simple shapes. Suppose one blob holds triangles of various sizes, and neighboring blobs hold circles and squares. Take a sample between the three and decode it. You get a triangle-like figure whose shape and color differ from all of them — yet it is a meaningful shape rather than a random one. Semantics survives the interpolation. Without regularity, two codes close together in latent space can decode to wildly different outputs; with regularity, decoded neighbors stay neighbors.

Real-world: language-model distillation pipelines literally minimize KL divergence between a big teacher model's output distribution and a smaller student's; policy-gradient methods in reinforcement learning constrain each update by its KL distance from the previous policy. The same directed measure you derived from a coin toss prices those systems' central trade-offs.

9.5 The Training Mathematics: Intractability and the ELBO

9.5.1 Formalizing the Decoder

Set up the notation cleanly. Recall the pipeline. Original data flows through the encoder, which yields either latent codes or, in the VAE, distribution parameters ${\mu }_Z$ and ${\Sigma }_Z$. A sample $z_i$ is drawn. The decoder converts $z_i$ into a reconstruction $x_i^{\prime }$. Training drives $x_i^{\prime }$ toward $x_i$ in the average sense.

Probabilistically, the decoder defines a conditional distribution:

$$p_{{\theta }^{\ast }}(x\mid z_i)$$

Here $z_i$ is the sampled latent value and $\theta$ — written with a star when we mean the true underlying parameters — collects the parameters of the decoder. The decoder learns the probability distribution of the reconstructed variable given the sampled latent. Concretely, for image data this conditional says: given this code, how likely is each possible output image? A common choice treats the output pixels as real values centered on the decoder's prediction, so high likelihood means the decoder confidently predicts what it renders.

Training then seeks the decoder parameters $\theta$ that maximize the likelihood of the training data under this model, $p_{\theta }(X)$. That sentence hides the entire difficulty of VAE training, and unpacking it fills the rest of this section.

9.5.2 Why Exact Likelihood Maximization Is Intractable

Write the marginal likelihood using the chain rule of probability — every data point can be produced from some latent value, so total likelihood sums (or integrates) over all ways production could have happened:

$$p_{\theta }(x)=\int p_{\theta }(z)p_{\theta }(x\mid z)dz$$

and in discrete form:

$$p_{\theta }(x)=\sum _z{p}_{\theta }(z)p_{\theta }(x\mid z)$$

The first factor is no problem: the prior is the assumed normal Gaussian, fully known, evaluated at any $z$ by plugging into the bell-curve formula. The second factor is computable too — pass any single $z$ through the decoder and read off its output density. The killer is quantity. Evaluating this sum demands passing every possible latent value $z_i$ through the decoder to collect its contribution. Too many values exist. With even a modest 200-dimensional continuous latent space there are uncountably many candidates; no loop enumerates them all. Direct maximization becomes intractable — not slow, but impossible to finish.

Bayes' rule offers a rearrangement, and it fails the same way. The posterior density — given the data, the probability of the latent — is:

$$p_{\theta }(z\mid x)=\frac{p_{\theta }(x\mid z)p_{\theta }(z)}{p_{\theta }(x)}$$

The posterior describes the density of a variable that depends on the original training data: which latent values could have been responsible for this image? The denominator contains $p_{\theta }(x)$ again — the same intractable marginal, now sitting in the one slot Bayes' rule requires you to fill. Rewrite however you like; the posterior stays intractable. Both doors out of the room are locked by the same key.

9.5.3 The Encoder Approximation

The escape is a second network. In addition to the decoder model $p_{\theta }(x\mid z)$, define an encoding distribution that approximates the posterior:

$$q_{\varphi }(z\mid x)\approx p_{\theta }(z\mid x)$$

This approximation network makes approximate maximization of the data likelihood possible, and the whole objective becomes optimizable with backpropagation. Summarizing the division of labor:

• Given $X$, the encoder learns the mean vector ${\mu }_{Z\mid x}$ of the latent distribution and its covariance matrix.
• The decoder — called the generator, because it generates the data — maps sampled latents to data.
• The encoder — called the inference network or recognition network, because it recognizes data vectors and infers their latents — maps data to distribution parameters.
• Both parameter sets, $\theta$ for decoding and $\varphi$ for inference, are learned jointly.

9.5.4 Dimension Bookkeeping: A Concrete Example

Fix numbers so the shapes stop being abstract. Let the input be a 32 by 32 image (so $D=1024$) and choose a latent dimension of $d=200$. Then:

• ${\mu }_{Z\mid x}$ is a 200-dimensional vector.
• ${\Sigma }_{Z\mid x}$ is a 200 by 200 covariance matrix.
• The noise draw $\epsilon \sim \mathcal{N}(0,I)$ is a 200-dimensional vector, since $I$ is the 200 by 200 identity whose diagonal entries are 1 and off-diagonal entries 0.
• The sampled latent is $z={\mu }_{Z\mid x}+{\Sigma }_{Z\mid x}\epsilon$. To be really correct about matrix-vector dimensions, the $200\times 200$ covariance multiplies the $200\times 1$ noise vector, producing another $200\times 1$ column, which adds elementwise to the $200\times 1$ mean.

Shape check on every product: $(200\times 200)(200\times 1)=200\times 1$, then $(200\times 1)+(200\times 1)=200\times 1$. Nothing ever leaves column form.

The sampled $z$ passes through the decoder, which produces the distribution $p_{\theta }(x\mid z)$ over decoder outputs; drawing from that distribution yields the actual reconstruction. Training adjusts $\theta$ and $\varphi$ together through backpropagation while the loss shrinks.

9.5.5 Deriving the Evidence Lower Bound Step by Step

Training maximizes the log-likelihood, averaged over the training set. Take one training point $x_i$ and follow every algebraic step. Two warnings before the symbols fly: first, this derivation is heavy symbol pushing — the payoff is not a new quantity but the discovery that VAE training maximizes a bound, never the likelihood itself. Second, nothing below approximates anything; every line is exact until Step 5, where an inequality is introduced deliberately.

Step 1 — wrap the target in an expectation. The log-likelihood does not depend on the latent, so averaging it over any latent distribution changes nothing:

$$\log p_{\theta }(x_i)={\mathbb{E}}_{z\sim q_{\varphi }(z\mid x_i)}[\log p_{\theta }(x_i)]$$

The expectation of a constant equals the constant — that is the entire content of this line. The latent values $z$ are drawn from the encoder distribution $q_{\varphi }(z\mid x_i)$; in practice each sampled $z$ arises by bringing in a random $\epsilon$ vector. Why bother writing a constant as an average? Because the next steps need the averaging machinery inside the brackets.

Step 2 — apply Bayes rule inside the logarithm. Expand the joint as conditional times prior, then divide by the posterior:

$$\log p_{\theta }(x_i)={\mathbb{E}}_{z\sim q_{\varphi }(z\mid x_i)}\left[\log \frac{p_{\theta }(x_i\mid z)p_{\theta }(z)}{p_{\theta }(z\mid x_i)}\right]$$

Nothing has changed yet — pure symbol pushing, exactly as announced. The fraction is just $p_{\theta }(x_i)$ rewritten via Bayes' rule, since $p(x\mid z)p(z)=p(x,z)$ and $p(x,z)/p(z\mid x)=p(x)$.

Step 3 — multiply by one. Insert the factor $q_{\varphi }(z\mid x_i)/q_{\varphi }(z\mid x_i)$, which equals 1, and regroup:

$$\log p_{\theta }(x_i)={\mathbb{E}}_{z\sim q_{\varphi }(z\mid x_i)}\left[\log \left(\frac{p_{\theta }(x_i\mid z)p_{\theta }(z)}{q_{\varphi }(z\mid x_i)}\cdot \frac{q_{\varphi }(z\mid x_i)}{p_{\theta }(z\mid x_i)}\right)\right]$$

Multiplying by 1 cannot change anything mathematically, but it changes everything structurally: the encoder distribution now appears twice, once in each fraction, preparing the split.

Step 4 — split the logarithm into three named pieces. A logarithm of a product splits into a sum of logarithms: $\log (a\cdot b)=\log a+\log b$. Apply that, then read each piece against the KL definition from Section 9.4 — weighting log-ratios by probabilities under a distribution. The numerator group $p_{\theta }(x_i\mid z)$ is the decoder transformation. The ratio $p_{\theta }(z)/q_{\varphi }(z\mid x_i)$ forms one divergence (negated). The leftover ratio $q_{\varphi }(z\mid x_i)/p_{\theta }(z\mid x_i)$ forms another:

$$\log p_{\theta }(x_i)=\underset{\text{reconstruction term}}{\underbrace{{\mathbb{E}}_{q_{\varphi }(z\mid x_i)}[\log p_{\theta }(x_i\mid z)]}}-\underset{\text{prior-matching term}}{\underbrace{D_{\mathrm{KL}}(q_{\varphi }(z\mid x_i)\Vert p_{\theta }(z))}}+\underset{\text{always }\ge 0}{\underbrace{D_{\mathrm{KL}}(q_{\varphi }(z\mid x_i)\Vert p_{\theta }(z\mid x_i))}}$$

Walk the sign bookkeeping once slowly. The split gives $\mathbb{E}[\log p_{\theta }(x_i\mid z)]+\mathbb{E}[\log \frac{p_{\theta }(z)}{q_{\varphi }(z\mid x_i)}]+\mathbb{E}[\log \frac{q_{\varphi }(z\mid x_i)}{p_{\theta }(z\mid x_i)}]$. Each expectation of a log-ratio under its own numerator's distribution is exactly a KL divergence, so the middle piece enters as $-D_{\mathrm{KL}}(q_{\varphi }\Vert p_{\theta }(z))$ and the last as $+D_{\mathrm{KL}}(q_{\varphi }\Vert p_{\theta }(z\mid x_i))$.

The first term rewards reconstructions the decoder assigns high likelihood. The second is a KL divergence pulling the encoder distribution toward the prior. The third is a KL divergence between the encoder distribution and the true posterior — and evaluating it directly is exactly the intractable problem identified above.

Step 5 — drop the impossible term using its sign. The last KL divergence is greater than or equal to 0 always — proven in Section 9.4 from the inequality $-\log y\ge 1-y$. So deleting it can only lower the right-hand side, leaving a lower bound on the log-likelihood:

$$\log p_{\theta }(x_i)\ge \underset{\text{ELBO}(x_i)}{\underbrace{{\mathbb{E}}_{q_{\varphi }(z\mid x_i)}[\log p_{\theta }(x_i\mid z)]-D_{\mathrm{KL}}(q_{\varphi }(z\mid x_i)\Vert p_{\theta }(z))}}$$

This bound is the variational lower bound, better known as the ELBO — the evidence lower bound, sometimes heard aloud as the elbow function. Since the discarded KL term is always non-negative, maximizing the ELBO pushes up a guaranteed floor under the log-likelihood. Maximization acts on both pieces at once. Raise the reconstruction term — equivalently, shrink the reconstruction error between original and generated data. And shrink the KL divergence, which pins the learned encoder distribution close to the normal Gaussian. That double duty is precisely the informal loss of Section 9.3, now derived rather than asserted. Exact likelihood maximization is out of reach; maximizing the likelihood lower bound is what VAE training actually does.

One more insight hides in the discarded term: the gap $\log p_{\theta }(x_i)-\text{ELBO}(x_i)$ equals exactly the intractable KL divergence between the approximation and the true posterior. Better encoders squeeze the gap shut; the bound gets tight precisely when $q_{\varphi }$ becomes the real posterior. So tracking the ELBO tracks both the model and the quality of its own approximation.

9.5.6 Why Sampling Is Needed for Backpropagation

Why insist on working with sampled latent values instead of the raw distribution parameters? Because optimization runs on gradients. Eventually you must differentiate the loss with respect to every parameter in $\theta$ and every parameter in $\varphi$, then adjust:

$$\theta \leftarrow \theta -\eta \frac{\partial L}{\partial \theta },\varphi \leftarrow \varphi -\eta \frac{\partial L}{\partial \varphi }$$

where $\eta$ is the learning rate — how big each correction step is. For the gradients to flow from the decoder back through to the encoder, a concrete value must traverse the whole path. The sampled latent provides that bridge: $z$ is nothing but $\mu$ plus scaled $\epsilon$. Writing it that way relocates the randomness to the network's input edge — the noise enters as data would, and the path from $\mu$ and $\Sigma$ to the loss runs entirely through ordinary deterministic operations a derivative can climb. This relocation is known as the reparameterization trick. As backpropagation nudges the parameters toward the local minimum, the $\mu$ values and the $\Sigma$ values update along with everything else. Freeze the distribution parameters out of the forward path — say, by emitting a bare sampled number with no visible $\mu$, $\Sigma$ ancestry — and the encoder receives no learning signal at all.

9.5.7 Mini-Batch Training and Generation After Training

Training follows the standard rhythm:

1. Split the training set into mini-batches.
2. Pass each batch through the encoder-decoder combination: encode to $\mu ,\Sigma$; draw $\epsilon$; sample $z$; decode; score the reconstruction plus KL terms.
3. Accumulate the loss over that batch and adjust $\theta$ and $\varphi$ once.
4. Repeat batch after batch until training stabilizes.

Each pass costs one encoder evaluation, one random draw, one decoder evaluation, and their backward sweeps per item — linear in batch size, with the KL term closed-form cheap because both distributions are Gaussians. The practical limit is the usual one: memory for activations during backprop.

When training is over, throw away the encoder. Generation needs none of it. Keep the decoder together with the learned machinery of mean vectors and covariances. Call a random number generator to produce $\epsilon$, and build multiple values of $z_i$ from it. Pass them to the decoder and harvest new data. Each draw is independent, and all draws run in parallel. The asymmetry is worth pausing on: training needed two networks and data; deployment needs one network and dice.

Real-world: anomaly detection systems score new inputs by their ELBO — a low bound means the model cannot reconstruct the item well, flagging it as unlike anything seen; semi-supervised pipelines reuse the inference network's codes as features for downstream classifiers, extracting double duty from the trained encoder before discarding it.

9.6 Generating Data with a Trained VAE

9.6.1 Sweeping Two Latent Dimensions: Worked Demo

Everything before this point built the machine; now watch it run.

9.6.2 Student Question: Using Generated Data for Machine Learning

The distinction deserves emphasis: classic augmentation rearranges existing examples — the same photo shifted two pixels left. VAE synthesis produces items that never existed, drawn from the learned distribution. Both fight overfitting; only the second expands what the model can plausibly see.

9.6.3 Choosing the Prior Distribution

The prior deserves one honest caveat. The normal Gaussian was assumed partly for convenience. With a Gaussian prior, computing the KL divergence stays simple, and so does updating the parameters of both encoder and decoder to reduce it — Gaussians come with closed-form distances between them. More complicated priors are legal — a mixture of Gaussians, for instance, several blobs instead of one. Practice has shown, though, that switching priors barely moves the quality of reconstruction or of new data. That observation is why the normal Gaussian remains the standard prior choice today: simplicity bought at zero measured cost is a good deal.

Real-world: attribute-controlled sampling turns face models into production tooling for recognition systems, and the same generated-data logic strengthens supervised training wherever usable data is scarce — medical imaging stands out, where collecting patient scans is slow and expensive but every extra plausible training case improves diagnostic models.

9.7 Applications, Strengths, and Limitations

9.7.1 Faces: Attribute Dimensions in the Latent Space

Train an input-decoder combination on mugshots of people. Once training finishes, pick out two latent dimensions and sweep samples across them. The decoder renders different versions of the same face. In one sweep a person who looked serious acquires a smile. That latent dimension is capturing smilingness — the training set contained other people smiling too, and the latent organization absorbed the concept.

Different regions of the latent space capture different high-level concepts, and which concepts appear depends on what the training images contain. Observed examples include male versus female faces, skin tone, smilingness, eyeglasses, beard, age, and head pose. Head pose means looking straight at the camera, turning toward a profile, or tilting the head back. Build a rich enough latent space from varied training images and you can sample along any of these abstract axes to create variations of a base image. Nobody labeled smile during training; the concept emerged because it helped reconstruction across many photos.

9.7.2 Face Recognition Robustness

Why does attribute variation matter operationally? Consider a recognition system for one individual. The database holds a single serious-looking photo taken around age 30. Twenty years later, investigators compare the person's current face against that stored face. Matching can fail outright — hairline moved, glasses added, expression hardened. It works only if the system was trained not just on the available facial data but on deliberate variations of it: aged renderings, altered poses, varied backgrounds.

A VAE manufactures exactly those variations. This is data augmentation done in a far smarter way than rescaling or rotating pixels. Bring in abstract latent dimensions — age, smilingness, skin tone, eyeglasses, beard — and synthesize changes along them. Feeding such augmented data to supervised training improves accuracy and prepares the recognizer for practical deployment.

9.7.3 Strengths and Weaknesses Summary

On the credit side, the VAE is a principled approach to generative modeling: a maximum-likelihood-based method. It permits inference of $q(z\mid x)$, so given data you can evaluate how likely it is under the model and judge whether generated output is plausible. It is good for feature representation — the feature size is far smaller than the original data, whereas a flow model's latent space matches the data dimension exactly.

On the debit side, training maximizes the lower bound of the likelihood, not the actual likelihood, because the actual likelihood is intractable. Samples from the original formulation come out blurrier than GAN output — GANs are covered next. They also trail sharp autoregressive generators such as PixelCNN or PixelSnail. Against those cons stands one more pro worth weighing: training is stable, with no jumping of the loss function up and down.

When to pick which: need calibrated probabilities or a reusable encoder — VAE territory; need maximally crisp samples and can absorb unstable training — GAN territory.

9.7.4 Latent Space Arithmetic

The structured latent space supports calculation, not just sampling — a practice known as latent space arithmetic. You can interpolate between a male voice and a female voice. Subtract the features of a piggy face from one image, then add the body of a pig to a cat image. The result lands on an intermediate concept between pig and cat. Add smiliness to a face. Add age to a person's portrait. Add eyeglasses. In the vector space of latents, concept directions behave like arithmetic operands, and editing data reduces to adding and subtracting along them.

Real-world: photo-editing suites expose "aging" and "smile" sliders powered by latent-direction arithmetic; voice-cloning products interpolate speaker latents between reference voices to hit target pitch and timbre.

9.8 Beta-VAE: Disentangled Representations

9.8.1 Motivation: Entangled Attributes

Practitioners noticed a flaw in plain VAE latents. The abstract dimensions — age, smile, skin tone — are not completely orthogonal. Increase the smile and the person's apparent age shifts too. Age the face and the skin tone drifts with it. The attributes are entangled: correlated directions rather than independent controls.

Picture an old mixing desk whose faders are linked by gears: push the treble up and the bass creeps along with it. You asked for one change and got two. A well-built desk has independent faders — each slider moves exactly its own channel. Plain VAE latents are the geared desk; editing applications need the independent one. Changing age should leave skin tone untouched; changing skin tone should leave gender appearance untouched. Achieving independent latent modification requires disentangled dimensions, and a variant built for exactly that purpose is the beta VAE, written β-VAE.

9.8.2 The Beta-Weighted Loss

The modification to the loss function is tiny. Multiply the KL divergence term by a coefficient $\beta$:

$$L_{\beta }={\mathbb{E}}_{q_{\varphi }(z\mid x)}[\log p_{\theta }(x\mid z)]-\beta D_{\mathrm{KL}}(q_{\varphi }(z\mid x)\Vert p(z))$$

Every symbol is inherited from Section 9.5: the expectation averages decoder log-likelihood under encoder samples, the divergence measures distance to the standard-normal prior $p(z)=\mathcal{N}(0,I)$, and only the new dial $\beta$ is added. With $\beta =1$ this is the standard VAE objective. Raise $\beta$ and the optimizer, chasing the enlarged penalty, drives the KL divergence even smaller.

Take $\beta =100$, for example. Minimizing the weighted objective then forces the divergence between the learned distribution and the normal Gaussian to become much smaller. The latent distribution gets pressed hard against the isotropic prior, in which $Z_1,Z_2,Z_3$ are completely uncorrelated. The learned covariance flattens toward diagonal — and a purely diagonal covariance means every dimension is disentangled from every other. Nothing in this argument needs the exact value 100; it only needs $\beta$ large enough that structuring the latent space becomes worth more to the optimizer than pixel-perfect reconstruction.

9.8.3 Capacity Constraint Interpretation and Results

The $\beta$ weighting is also known as the capacity constraint: larger beta spends more of the model's capacity on structuring the latent space and less on pixel-perfect reconstruction. The name reads directly off the formula — the KL term caps how much information each code may carry about its input, and $\beta$ sets how tightly the cap clamps.

The payoff shows in controlled edits:

• Skin color changes while expression, pose, and apparent age hold steady.
• On chairs, leg style, width, and viewing azimuth can each move independently; viewing azimuth is how the object sits oriented toward the viewer.
• New chair variants emerge feature by feature rather than all at once.

Editing proceeds latent-feature by latent-feature instead of collectively, which is the entire design goal of disentanglement. The geared mixing desk gets its sliders separated: one fader per attribute, no cross-talk.

9.9 VQ-VAE: Vector-Quantized Variational Autoencoder

9.9.1 Motivation: Discrete Latent Spaces

The original VAE assumes a continuous latent distribution — the normal Gaussian is a creature of continuous random variables, able to take any value on the line or plane. Many practical domains refuse to be continuous. When the meaningful content is a finite menu of categories, formulating generation in discrete space fits better and can render new data more faithfully. The fix is a variant called VQ-VAE, where VQ stands for vector quantized.

9.9.2 Where Discrete Codes Fit: Images, Language, Audio

Take an image containing a background and an object. Describe it discretely: the shape of the object, its color, its orientation, the kind of background, the texture of that background, the lighting. None of these is a continuous dial. Color might admit perhaps 20 possible values and no more. Texture outdoors might span 5 or 10 types. A fixed menu of possibilities covers the scene.

The same reasoning extends beyond images — the argument holds for one-dimensional data too:

• Language: tokens are categorical by nature; the vocabulary is a finite list.
• Audio: the descriptors are discrete — which language is being spoken, male or female voice, the speaker's age bracket, the topic of discussion.

Finite, countable, categorical. For data best represented in discrete fashion, VQ-VAE is the formulated tool.

9.9.3 Codebook Quantization Mechanism

Replace the continuous latent space with a discrete embedding space holding $K$ discrete latent variables, written $e_1$ through $e_K$. Each $e_i$ is itself a real-valued vector — say 200-dimensional — but only these $K$ fixed vectors may serve as codes; infinitely many values are not permitted.

The mechanism runs in three moves:

1. Encode input $x$ into a continuous latent $z_e(x)$ — a 200-dimensional vector in our running example.
2. Compare $z_e(x)$ against all $K$ codebook vectors and record the index $i$ minimizing the distance:

$$i=\underset{j\in \{1,\dots ,K\}}{\mathrm{argmin}}\Vert z_e(x)-e_j\Vert$$

The $\arg \min$ notation means: among all indices $j$ from 1 to $K$, return the one whose codebook vector sits closest to the encoder output under the Euclidean norm.

1. During decoding, pass the selected fixed embedding $e_i$ — not the raw $z_e$ — through the decoder, then compute reconstruction losses as usual.

So a given latent value maps to a sequence of indices. What the encoder produces is thereby restricted to a small set of possibilities called the codes, and the collection of embeddings is the codebook.

9.9.4 The Three-Part Loss and Stop-Gradient

VQ-VAE training maximizes the log-likelihood of the generated data given the quantized latent $z_q$. The subscript records that the value passed downstream is the selected codebook entry, not the encoder's raw output. On top of the likelihood term sit two more loss components, and they pull in opposite directions:

• Codebook alignment: move each embedding $e_i$ closer to the encoder outputs $z_e$. Otherwise encoder outputs would be jerked around by arbitrary random code vectors — the codebook starts random, and left alone it would steer generation badly.
• Commitment: move the encoder outputs closer to their assigned embeddings. The process is two-way — codebooks approach encodings, and encodings commit to codebooks.

Assembled as one objective to minimize:

$$L=\underset{\text{reconstruction}}{\underbrace{-\log p(x\mid z_q(z_e))}}+\underset{\text{codebook alignment}}{\underbrace{\Vert \mathrm{sg}\left[z_e\right]-e{\Vert }^2}}+\underset{\text{commitment}}{\underbrace{{\beta }_{\text{commit}}\Vert z_e-\mathrm{sg}\left[e\right]{\Vert }^2}}$$

This matches the canonical published form of the VQ-VAE objective exactly: the alignment term carries the stop-gradient on the encoder output so it updates only the codebook, and the commitment term carries it on the embedding so it updates only the encoder. Here $\mathrm{sg}[\cdot ]$ is the stop-gradient operator — during the forward pass it passes its argument through unchanged, but during backpropagation it blocks gradients from flowing through, letting exactly one side of each squared term receive learning signal. In the alignment term the gradient reaches $e$ but not $z_e$; in the commitment term it reaches $z_e$ but not $e$. The full treatment of training VQ-VAE, including this operator, opens the next session.

9.9.5 Resources and Next Steps

Two recommended VQ-VAE blogs were flagged as particularly good reading, with links traveling with the shared materials — going through them is strongly advised before continuing. The next session spends 15 to 20 minutes finishing VQ-VAE training, including the stop-gradient mechanics introduced above. It then introduces VQ-VAE-2, which stacks multiple layers of encoders and multiple layers of decoders; that hierarchy renders data at quality as high as competing methods or higher. After that begins the GAN — the generative adversarial network — scheduled as a multi-session deep treatment.

Exam Guidance Summary

• Exam relevance flag: the VAE is the autoencoder type explicitly designated for examinations, spanning the mid-semester and end-term scope. Treat this topic as core testable material, not enrichment.
• Expect conceptual questions on the design distinctions. Those cover why the latent distribution is learned instead of latent codes (distribution parameters grade a neighborhood of codes rather than a single point), why the Gaussian prior regularizes the space (it merges isolated code islands into smooth coverage), and what loss of semantics means (nearby codes decoding to unrelated outputs). Use the animal/vehicle and shape/color illustrations as reference examples.
• Expect mathematical questions on the entropy formula and its extremes: 0 for deterministic systems, 1 bit for the uniform coin toss. Also expect the significance of the minus sign ($\log p\le 0$ makes each term negative; flipping keeps entropy positive), the KL divergence definition ${\sum }_{x}P(x)\log \frac{P(x)}{Q(x)}$, and its properties: always non-negative, asymmetric — so not a metric distance. Then comes the step-by-step ELBO derivation, including why the final KL term can be dropped: it is always non-negative, so removing it leaves a valid lower bound.
• Know the sampling mechanics cold: $z=\mu +\Sigma \epsilon$ with correct matrix dimensions — a $d\times d$ covariance multiplying a $d\times 1$ noise vector adding to a $d\times 1$ mean. Know why sampling enables gradient flow to the encoder (the reparameterization relocates randomness to the input edge), and know the update rules $\theta \leftarrow \theta -\eta \partial L/\partial \theta$, $\varphi \leftarrow \varphi -\eta \partial L/\partial \varphi$.
• Study advice: work through the offline reference list shared for this topic. Read the two recommended VQ-VAE blogs before the next session, since VQ-VAE training continues next time and the stop-gradient discussion builds on them.
• Sequencing signal: the next topic, GANs, receives a five-to-six-hour allocation. Budget preparation effort accordingly, and expect the remaining VQ-VAE and VQ-VAE-2 material to be examined only after it is completed.

Key Industry Applications

• Conversational AI systems: ChatGPT and Claude exemplify autoregressive, token-by-token generation — the slow-but-stable paradigm the VAE's parallel one-shot generation contrasts with. Every chat reply you wait for is the sequential-generation cost paid in production.
• Synthetic training data: VAE-generated samples augment scarce datasets for supervised learning, extending classic augmentation (scaling, translation, rotation) with new synthesized instances; the whole pipeline is unsupervised. Medical imaging and industrial defect inspection lean on this where collecting real examples is slow or costly.
• Face recognition robustness: attribute-controlled synthesis — age progression, pose, smile, eyeglasses, beard, skin tone, background — hardens recognizers against real-world variation. One example: matching a decades-old mugshot to a current face.
• Content creation and marketing: generated imagery supports brochures, creative assets, and question-answering media pipelines.
• Image generation stack: VAEs and GANs form the classical backbone of image generative AI. Autoregressive image models such as PixelCNN and PixelSnail set the sharpness benchmark that early VAEs chased. VQ-VAE targets discrete, categorical structure in images, language, and audio — the same discrete-token logic that modern codec-style generators exploit. VQ-VAE-2's hierarchical encoders and decoders push render quality higher.