Discrete Latent Spaces and Generative Adversarial Networks

This session closes the discrete-latent branch of variational autoencoders — VQ-VAE and its hierarchical successor VQ-VAE-2 — and then opens a completely new family of generative models: generative adversarial networks (GANs). GANs matter because they break the pattern every earlier method followed: instead of writing down and maximizing a probability distribution, they learn to generate through an implicit contest between two networks. We build the original 2014 formulation piece by piece, read its famous minimax objective term by term, walk its training loop step by step, and finish with the two scores — the Inception Score and the Fréchet Inception Distance — used in practice to judge what such models produce.

Here is the road map for the session:

1. Recap and motivation (10.1–10.2). A quick refresher on the standard VAE and beta-VAE, and why categorical data calls for a discrete latent space.
2. VQ-VAE (10.3–10.4). The vector-quantized VAE: a codebook of discrete codes, a three-piece loss, the stop-gradient trick that makes it trainable, and how a PixelCNN prior over codes unlocks sampling.
3. VQ-VAE-2 (10.5). The hierarchical version that generates noticeably sharper images by separating coarse structure from fine detail.
4. GANs (10.6–10.8). Implicit generative modeling: generator versus discriminator, the minimax objective, the training loop, and the non-saturating trick.
5. Evaluation (10.9–10.10). How to score generated images with the Inception Score and the Fréchet Inception Distance — and why you should always read them together.

10.1 Recap of Variational Autoencoders and Why Discrete Latent Spaces Matter

10.1.1 How the Standard VAE Works and Generates

Recall from the previous session how a variational autoencoder (VAE) works. The original formulation assumes the latent space is continuous: each latent coordinate can take any real value, and nearby values decode to similar-looking data.

Given an input, we build an autoencoder with a variational regularizer in place so that the encoder does not output the inputs as-is; it learns the parameters of an underlying distribution over the latent representation. Concretely, the encoder outputs two things per input:

• $\mu$ (mu) — the mean vector, the "center point" of where this image sits in latent space;
• $\Sigma$ (sigma) — the covariance matrix, which describes how spread out that placement is.

The variational regularizer then forces the distribution the encoder learns to stay very close to a standard Gaussian prior in the latent space: zero mean and unit covariance, with ones on all diagonal entries. During training, the deviation of the learned distributions from this standard normal is minimized by adjusting the parameters of the encoder. In the objective this regularizer appears as the Kullback–Leibler (KL) divergence term between the encoder distribution $q_{\varphi }(z\mid x)$ and the prior:

$${\mathcal{L}}_{\text{VAE}}=\underset{\text{reconstruction}}{\underbrace{{\mathbb{E}}_{q_{\varphi }(z\mid x)}[-\log p_{\theta }(x\mid z)]}}+\underset{\text{regularizer toward }\mathcal{N}(0,I)}{\underbrace{D_{\mathrm{KL}}(q_{\varphi }(z\mid x)\Vert \mathcal{N}(0,I))}}$$

Here $x$ is a training input, $z$ is the latent vector, $q_{\varphi }(z\mid x)$ is the encoder's approximate posterior (the Gaussian whose mean and covariance the encoder predicts), and $p_{\theta }(x\mid z)$ is the decoder's likelihood of the data given a latent code. Minimizing the first term asks the decoder to reconstruct the input faithfully; minimizing the second keeps every encoded distribution close to the standard normal. This is exactly the evidence lower bound (ELBO) machinery covered earlier — here we only need the shape of it, not the full derivation.

Once training is over, generation is simple: sample a latent vector from the standard normal prior, push the sampled values through the decoder, and semantically meaningful new samples come out. That is the basic principle of the standard VAE.

One assumption deserves attention before we move on: continuity itself.

10.1.2 Categorical Descriptions Call for a Discrete Latent Space

So here is the question that motivates this topic: should we restrict all VAE discussion to continuous latent spaces? In many scenarios you can associate discrete labels with training images — these labels are descriptions of the data. Examples:

• whether an image was taken indoors or outdoors;
• whether it includes mechanical objects or face images;
• for an image of a person: male or female, facing the camera or not.

All of these are categorical pieces of information about the image. Given that data so often carries such discrete descriptions, it makes sense that the latent space itself should be representable using discrete concepts. A picture is then summarized by a short list of tokens — much like how a caption summarizes a scene in words — instead of a long list of decimal numbers.

That idea produces variants of the original VAE known as VQ-VAE and VQ-VAE-2, which is where we pick up. The plan: keep the encoder–decoder skeleton of the VAE, but replace the "sample a Gaussian" middle with "look up the nearest entry in a learned dictionary of codes."

10.2 A Quick Word on Beta-VAE Before Moving On

10.2.1 Editing One Factor While Freezing the Rest

Beta-VAE is another variant of the original VAE, already covered adequately earlier, so only the essential idea is repeated here. It targets disentangled feature spaces: when you modify images along one factor, the other factors must stay put.

Suppose you want to edit a facial image — change a neutral expression into a very happy one — without inadvertently changing other attributes like the age of the individual or the skin tone. In an entangled space, moving along any single latent direction tends to rotate several attributes at once, so "make happier" also quietly makes the face older or lighter. Beta-VAE achieves the clean-slider behavior by adding extra weights to the KL divergence term of the VAE objective:

$${\mathcal{L}}_{\beta }=\underset{\text{reconstruction}}{\underbrace{{\mathbb{E}}_{q_{\varphi }(z\mid x)}[-\log p_{\theta }(x\mid z)]}}+\underset{\text{weighted regularizer}}{\underbrace{\beta D_{\mathrm{KL}}(q_{\varphi }(z\mid x)\Vert \mathcal{N}(0,I))}},\beta >1$$

The symbols are the same as in Section 10.1; the only new object is $\beta$ (beta), a constant that scales how strongly each encoded distribution is pushed toward the standard normal. With $\beta =1$ you recover the plain VAE loss. Raising $\beta$ squeezes the posterior distributions harder toward a sphere of independent, unit-variance axes — and strongly oriented axes are exactly what makes individual coordinates correspond to individual, interpretable factors rather than entangled diagonals.

A useful mental model: think of the ideal latent space as a mixing desk with one slider per attribute. Turning the "expression" slider up should move the image along the happy axis and touch nothing else. Beta-VAE's heavier regularization is a way of demanding that the learned axes line up with those sliders. The analogy breaks where real attributes are correlated in the data itself — if smiling people in your dataset happen to be younger on average, no axis arrangement can fully separate the two.

Keep this "edit one factor, freeze the rest" picture in mind; it is the same desire for interpretable structure that drives the discrete-latent work next.

10.3 VQ-VAE: Vector-Quantized Variational Autoencoder

10.3.1 From Image to Feature Grid

The starting point of VQ-VAE (the VQ stands for vector quantized) is ordinary convolutional encoding — exactly the kind of encoder you already know from CNNs. During encoding, the original image gets broken into blocks, and each block ends up with a high-dimensional feature value attached to it. Instead of the original RGB image, after the encoder has run, the encoded data lives as a grid of feature vectors.

A concrete example with shapes spelled out: start with a $256\times 256\times 3$ RGB image — that is $256\times 256$ pixels, each with red, green, and blue channels. Encode it with convolutional layers, exactly as in a CNN setting, until the image is represented on a $32\times 32$ grid where each of those grid cells carries a 128-dimensional feature vector. Each stride-2 downsampling stage halves the spatial size ($256\to 128\to 64\to 32$), while the channel count grows. So the encoder's output is not a single vector but a spatial array of vectors: $32\times 32=1024$ cells, each holding a vector $z_e\in {\mathbb{R}}^{128}$.

Picture the result as a shrunken, abstract version of the photo: same rough layout, but instead of raw colors, every position stores a 128-number description of "what content lives around here." Everything that follows operates on this array.

10.3.2 The Codebook and Nearest-Neighbor Quantization

Inside the VQ-VAE encoder sits a codebook: a collection of $K$ codebook vectors, each of the same dimension as the encoder features — 128 in our running example. Two things stay fixed throughout training: the number of codebook entries $K$ and their dimension. What is learned is their content. At initialization you may attach random feature values to each codebook entry.

Quantization then proceeds per grid cell, and it is simply a nearest-neighbor search:

1. Take one cell's feature vector $z_e$.
2. Compare it against everything in the codebook using a distance such as squared Euclidean distance.
3. Find whichever codebook entry is closest; record that entry's index.

Repeat for every grid point, so each position in the $32\times 32$ grid now points at some codebook index. Here is the key move: instead of passing the encoder's raw feature maps onward, you pass the corresponding codebook vectors into the decoder. If cell one matched codebook entry 1, cell two matched entry 53, cell three matched entry 2, and so on, the decoder receives the sequence $e_1,e_{53},e_2,\dots$ — actual stored codeword vectors, not encoder outputs. These go through the reverse of what happens inside a CNN and produce a reconstructed image.

The reconstructed image is compared against the original data, and a loss drives adjustments of the encoder and decoder parameters so the reconstruction error goes down. But reconstruction alone is not the whole story — the codebooks themselves also get updated during training, which leads directly to the loss function.

10.3.3 The Three-Piece Loss Function

Training a VQ-VAE minimizes a loss made of three distinct pieces. Reconstructed from the description, with the plain-language reading alongside:

$${\mathcal{L}}_{\text{VQ-VAE}}=\underset{\text{reconstruction}}{\underbrace{-\log p\left(x\mid z_q(x)\right)}}+\underset{\text{codebook loss}}{\underbrace{{\Vert \mathrm{sg}\left[z_e(x)\right]-e\Vert }_2^2}}+\underset{\text{commitment loss}}{\underbrace{\beta {\Vert z_e(x)-\mathrm{sg}\left[e\right]\Vert }_2^2}}$$

Read the symbols first. $x$ is the input image. $z_e(x)$ is the encoder output — the unquantized feature vector coming out of the encoding process for a given cell. $z_q(x)$ is the quantized version of that vector: the codebook entry $e$ that was selected as its nearest match. $\mathrm{sg}[\cdot ]$ denotes the stop-gradient operator, explained below. Finally, $\beta$ is a constant that sets how strictly you want the encoded values to commit to the codebook values. This is the standard formulation of the method: the stop gradients sit exactly where shown — frozen encoder output inside the codebook term, frozen codeword inside the commitment term.

Piece by piece, in the order described:

• Reconstruction term. The decoder models the likelihood of the data given the quantized codes, so the natural objective is to maximize $\log p(x\mid z_q(x))$. Since $p(x\mid z_q(x))$ is a probability between 0 and 1, its logarithm is negative; taking the minus turns it into something to minimize. Minimizing $-\log p(x\mid z_q(x))$ is minimizing the reconstruction error.
• Codebook loss. This measures how far each current codebook vector sits from the encoder outputs assigned to it. "Corresponding" means: if entry $e_3$ was the closest codeword for a particular grid cell, the loss for that cell is the difference vector between the third codebook vector and that cell's feature vector. You want to reduce these distances summed over the entire training dataset. As this term shrinks, the codebook contents migrate toward the values the encoder emits — the goal is to make the codebooks very similar to what comes out of the encoder.
• Commitment loss. The two-way partner of the previous term: while codebooks move toward encoder outputs, the encoder must also commit to codebook values. This term is the level of commitment the encoded values show toward the chosen codebook entries. It pulls $z_e(x)$ toward $e$ so both sides meet, and $\beta$ emphasizes that you want the commitment to hold quite strictly.

Notice the elegant symmetry: both auxiliary terms measure the same gap between $z_e(x)$ and $e$. They differ only in which side is frozen — and so in which parameters receive gradient updates. That is precisely what the stop-gradient notation encodes, and Section 10.3.4 unpacks it fully.

10.3.4 The Non-Differentiable Step and Stop Gradients

Here is the subtle trap in this loop. Somewhere inside it sits an operation that is not differentiable: the comparison-and-pick. You compare a feature vector against all the $e_i$, identify the index whose content matches best, and pass that particular codebook content to the decoder. Picking the index that gives the minimum is an argmin operation, and backpropagation cannot flow through it. Nobody asked about this, and it is exactly the right question to ask about this training loop.

Why does argmin kill gradients? Look at the toy example again. Nudge $z_e$ from $(0.9,-0.1)$ to $(0.92,-0.11)$: the distances reshuffle slightly, yet the winner stays $e_1$ and the decoder input stays exactly $(1.0,0.0)$. The output did not change at all, so the local rate of change — the derivative — is zero. Only when a nudge grows large enough to flip the winning index does the output jump, and jumps have no well-defined slope either. The quantized output is a step function of the encoder output: flat almost everywhere, discontinuous at decision boundaries. Gradients are the currency of backpropagation, and step functions pay nothing.

The fix is an engineering act built on the stop gradient operator, written $\mathrm{sg}[\cdot ]$. Its meaning: do not let gradient information flow past this point. Because the decoder inputs are only ever codebook contents — never the raw encoder outputs — you compute the loss gradient with respect to the decoder input, $\partial \mathcal{L}/\partial z_q$, and simply copy that value onto the encoder path:

$$\frac{\partial \mathcal{L}}{\partial z_e(x)}:=\frac{\partial \mathcal{L}}{\partial z_q(x)}$$

That is, treat the quantizer as if it were the identity during the backward pass: copy the partial derivative up to that point and then backpropagate as usual into the encoder weights. This is precisely why the notation writes a stop gradient on $z_q$. The forward pass uses the true quantized values; only the backward pass pretends the lookup never happened.

The same trick disciplines the two auxiliary terms:

• In the codebook loss, $\mathrm{sg}[z_e(x)]$ freezes the encoder side — within this term you update only $e$, moving codebook vectors toward the encoder outputs, and you do not touch $z_e$ here.
• In the commitment loss, $\mathrm{sg}[e]$ freezes the codebook — here you update $z_e$ by changing the encoder's parameters, and you deliberately do not change $e$.

Both things happen together: make the codebooks similar to what the encoder produces, and make the encoder outputs stick close to the codebooks. Each stop gradient protects exactly one player from being tugged by the term aimed at the other.

10.3.5 Updating the Codebook: Dictionary Learning

How, mechanically, does a codebook entry improve? Through dictionary learning: collect all the encoder output vectors $z_e$ for which this particular index was the closest match, and move the codebook vector to their mean:

$$e_j\leftarrow \underset{i:j=\arg {\min }_k\Vert z_e(x_i)-e_k\Vert }{\mathrm{mean}}{z}_e(x_i)$$

In words: a codebook vector becomes the average of all the encoded vectors that respond to it. Read the subscript carefully — the average runs over exactly those training cells whose nearest entry is $j$, and nobody else. Entries that win no votes stay put (or drift only via other mechanisms), which hints at a practical worry revisited in the pitfalls below.

That completes the picture of VQ-VAE training. A quick cost note rounds out the procedure: the nearest-neighbor scan costs $O(K\cdot D)$ per grid cell per pass (with $D=128$ and, say, $K=512$ entries in realistic setups), multiplied by 1024 cells per image. Exact scans are fine at lecture scale; production systems reach for approximate nearest-neighbor indexes when $K$ grows large. Alternatives worth knowing: k-means-style clustering plays the same role offline, and straight-through estimation is the standard substitute whenever a hard choice sits inside an otherwise differentiable network.

10.3.6 Student Questions and Answers

A second question probed where each piece of machinery lives.

And the sharpest question of the day — the one nobody asked, so it was raised in class:

10.3.7 Applications Beyond Images

Real-world: VQ-VAE has been applied very successfully not only to images but to one-dimensional signals such as audio. It performs well on ImageNet-scale reconstruction, and it is genuinely useful for speech modeling: converting male speech to a female voice or vice versa, changing speech from one male speaker to another male speaker — various forms of voice style transfer are possible. The reason the discrete codes suit speech so well is that speech itself decomposes into a small inventory of reusable units (phonemes and their acoustic variants), which a codebook represents naturally.

In the speech setting the decoder is typically a WaveNet, the autoregressive (AR) model architecture learned earlier for raw audio: the quantized codes carry the content, and WaveNet renders them into waveform samples. The same recipe transfers to music and other time-series signals — anywhere a finite vocabulary of prototypes is a natural description.

10.4 Generating New Samples from a Trained VQ-VAE

10.4.1 Sampling in a VAE versus Sampling in a VQ-VAE

Refresh how sampling worked in the standard VAE: draw a random number $ϵ$ and form the latent sample as

$$z=\mu +\sigma \odot ϵ$$

where $\mu$ is the learned mean vector, $\sigma$ the learned scale, and $ϵ$ a noise vector whose dimension equals the dimensionality of the latent space. One clarification on where $ϵ$ comes from: in the standard formulation each component of $ϵ$ is drawn from a standard normal distribution $\mathcal{N}(0,1)$, not from the interval between zero and one, so the sampled $z$ spreads symmetrically around the mean $\mu$ with spread set by $\sigma$; the product $\sigma \odot ϵ$ multiplies elementwise. That single closed-form expression is the whole sampler — the prior is known, so one draw suffices.

VQ-VAE cannot work this way. Its latent objects are discrete index grids, and there is no formula that conjures a plausible grid out of thin air. The working assumption during training was that the codes spread roughly uniformly across the codebook: if you use, say, 30 codebook entries, then maybe one-thirtieth of all encoded blocks respond to entry 1, a similar number to entry 2, and so on down to entry $i$. But uniform guessing is exactly wrong for generation: real images favor particular combinations of neighboring codes — edges pair with edges, fur pairs with faces. To produce valid new signals you need a reasonable prior over the discrete codes, one learned from the data.

10.4.2 Learning a Discrete Prior with PixelCNN

After training finishes, every training image has left behind a short sequence of codebook indices. Picture a case with $2\times 2$ blocks per image: one image might have produced indices 11, 3, 3, 30 — four cells, four indices; another image might have produced 9, 12, 7, 23. Across the whole dataset you accumulate many such discrete sequences — and these are pure discrete data you can model.

So fit a probability distribution over them, exactly the kind of autoregressive distribution learning done with PixelCNN in the AR-model part of the course: learn a PixelCNN-style distribution estimate over the coded inputs — "coded" here means represented purely as codebook indices. The autoregressive trick predicts each index from all the ones before it in raster order, which is precisely the kind of dependency structure (edges near edges, textures repeating) that makes some index grids plausible and most others nonsense. Once that distribution is learned, use PixelCNN to sample new index sequences. Walk each sampled sequence to the codebook, pick up the corresponding entries, and pass them through the decoder: out comes new data.

Real-world note on why one-dimensional signals are even friendlier here: the notion of temporal order makes their autoregressiveness natural. For images you impose an ordering by convention — given a particular block, all blocks on the same row to its left, plus every block in rows above, count as "past" data used to predict the current block. Audio needs no such convention; time provides it. This is also why WaveNet-style decoders and PixelCNN-style priors slot into the audio pipeline without rethinking.

10.5 VQ-VAE-2: Hierarchy for Sharper Generation

10.5.1 Two Encoders, Two Codebooks, Two Decoders

VQ-VAE-2 generates data with noticeably better clarity than flat VQ-VAE — good enough to compete strongly with images produced by generative adversarial networks. Where VQ-VAE beats a standard VAE is exactly where discreteness makes sense; VQ-VAE-2 improves on VQ-VAE. Its idea: don't use one encoder — bring a hierarchical structure into the encode-decode process.

Concretely, suppose the training set is a collection of $256\times 256$ images. The first encoder — call it the bottom encoder — compresses each image to a $64\times 64$ representation, each block perhaps carrying 64-dimensional features (illustrative numbers). Those encoded features feed a second encoder — the top encoder — which encodes further, to $32\times 32$ blocks. Typically the top level uses richer feature maps than the bottom: if the bottom maps are, say, 64- to 80-dimensional, the top might be around 160-dimensional. The intuition behind richer top features: fewer cells must cover more image, so each cell has to describe a larger, more abstract chunk of scene.

Both levels get their own codebook, applying exactly the principles from Section 10.3. At the top level, the training goal is that the top decoder's reconstruction should closely reproduce the $64\times 64$ encoded values the bottom encoder produced. Then, based on what flows out of the top decoder together with the bottom-level encodings, the lower-level codebook is formed. The key conditioning: the encoded values at the bottom layer are conditioned on the values generated by the top decoder. In other words, fine detail is never chosen in isolation — it is always chosen given the coarse layout above it.

Nothing prevents extending to three or four levels of encoders and decoders — the setup shown used three. But note the trade-off: the hierarchical structure makes training quite difficult, while the clarity of reconstructions and of newly generated data becomes very good, competitive with GAN-based techniques.

10.5.2 The Painting Analogy

Strip away the encoder-decode nomenclature for a moment and think in everyday terms — this is remarkably close to how humans work. Suppose you want to paint a nice cottage with a river and trees in the background. You probably would not start with the door handle. First you sketch the outline: the entrance to the house, the cottage shape, the areas reserved for greenery, the region where the river will run. Only after the outline exists do you step inside and add lower-level detail — where the door goes, where the windows are — then texture for the river, texture within the green region, texture on the entrance way.

Map that onto the architecture: the top-level encoder and decoder correspond to the outlines — the higher-order concepts — while the bottom level adds the low-level details, and those details are added conditioned on the high-level structure already drawn. Lower-level choices depend on the higher-level lines laid down first. This is analogous to how humans process information in multiple layers of abstraction, and it is the intuition for why VQ-VAE-2 works.

Where does the analogy break? A human painter can revise earlier strokes when details demand it; the feed-forward hierarchy cannot — once the top-level sketch is committed as codes, the lower levels live within it. That rigidity is also the point: it forces a clean division between planning and rendering.

10.5.3 Reconstruction Quality Levels and Sampling with PixelSNAIL

The quality ladder is easy to see in the reconstructions. From an original image, the top-level encoder–decoder combination alone yields an image noticeably less faithful — global shapes present, texture gone. Add the middle encoder — in the setup shown there were three encoders — and much of the graininess resolves: teeth become clear, eyes gain resolution. Use top plus middle plus bottom together and quality improves again. Read the ladder as an ablation study: each extra level contributes exactly the scale of detail it is responsible for, which is strong evidence the hierarchy really has separated coarse from fine.

For generating new data, you need probability distributions over the codebook usage at every level. Flat VQ-VAE used PixelCNN as its distribution estimator; VQ-VAE-2 uses PixelSNAIL, a higher-quality CNN-based distribution model with attention-style receptive fields that capture long-range dependencies between distant patches. The sampling order respects the hierarchy: start with the top-level codebook sequences and build the top prior; then, conditioned on the sampled top codes, learn/sample the bottom-layer codes. Decoding the full stack gives the finished sample.

Pull the family together:

Standard VAE — continuous latent space; beta-VAE — disentangled feature modification; VQ-VAE — discrete latent space; VQ-VAE-2 — a hierarchical structure that keeps the discrete latent space but layers it. The upcoming assignment lets you play with these alongside some GAN aspects.

Real-world: VQ-VAE-2's high-fidelity samples on large image datasets showed that discrete latent models could stand next to adversarial approaches on image quality — a result that later fed into architectures combining discrete codes with transformers for even larger-scale generation.

10.6 Generative Adversarial Networks: Learning Distributions Without a Likelihood

10.6.1 The Likelihood-Based Family So Far

GANs — invented in 2014 by Goodfellow, who was also the first author of the reference book on deep learning used throughout this course, one of the first books to cover the field broadly — differ from everything learned so far in one fundamental way. Every distribution-learning formulation in this course so far rests, to a large extent, on maximization of likelihood:

• Autoregressive (AR) models provided exact calculation of probability distributions — no approximation. Learn the distribution from data, sample it to create new data. Two costs: sampling is sequential (pixel 1, then pixel 2 given pixel 1, and so on), so generation is slow; and there is no concept of a latent space.
• Flow models associate a latent space with the data, giving the latent a deeper meaning than the raw pixels. But that latent space has the same dimension as the data — no compression ability — and the transformation must be bijective: two things cannot map to one during the transformation, because only invertible maps let you compute the exact likelihood by change of variables. Exact likelihood, though.
• VAEs also maximize probability, but only as an approximation: the exact likelihood is intractable because evaluating the KL divergence between the encoder and decoder transformations cannot be done. So we maximize the ELBO — the evidence lower bound — dropping the intractable positive KL term, and instead minimize the KL divergence between the approximate posterior and the standard normal prior while maximizing the data likelihood.

All of these rely on explicit calculation of the probability distribution. Whatever their differences, each one can answer the question "how likely is this point under my model?"

10.6.2 The Implicit Alternative and Its Trade-offs

A GAN takes a different route entirely: implicit modeling. It never explicitly calculates the probability distribution. What does that mean? Start from a random vector — usually called $z$, often 100-dimensional — pass it through a module called the generator, and out comes generated data. The generator's parameters are tuned until its outputs look like they came from the training distribution; the distribution itself lives implicitly in those weights, the way a forger's skill lives in their hands rather than in a written recipe.

Flow models share the skeleton (sample $z$, transform): there $z$ has the data's own dimension and you apply a learned inverse transformation; in a standard VAE you sample $z=\mu +\sigma ϵ$ from encoder-learned parameters and decode. The GAN starts from a small-dimensional vector and uses its generation network to create the data, but approaches the distribution implicitly rather than fitting it directly.

Two consequences follow, one pleasant and one awkward. Pleasant: data generation is completely parallel — unlike the AR sampler's sequential crawl, all of an image appears at once, so generation is fast. Awkward: there is no inference capability — no way to compute how likely a generated datum is. The likelihood calculation is absent, which is why evaluation of GANs needs its own treatment later in this session.

Worth fixing in mind, the checklist of things that make any generative model good:

1. Sampling — you can draw from it and create new data;
2. Likelihood evaluation — it can tell you whether generated data is plausible or not at all likely;
3. Trainability — optimization behaves well enough to run;
4. Representation — it offers something like a latent space.

GANs score brilliantly on parallel sampling and trainability, give up explicit likelihood scoring, and — depending on variant — provide representations.

Real-world context on momentum: the visible quality of GAN samples climbed dramatically — in 2014 outputs were black-and-white only; by 2015–16 modest-resolution color appeared; from 2021 onward, very high-quality, large, high-resolution images became routine. The topic is enormously broad — dedicated short courses of six to eight weeks exist on GANs alone — and heavily researched, with many variants producing excellent results. The roadmap: basic principles now; evaluation of GANs; a brief look at theory; then a fully convolutional implementation known as DCGAN; subsequent sessions cover the major variants and address training instability.

10.7 Inside a GAN: Generator versus Discriminator

A helpful everyday picture before the architecture: imagine a forger trying to create fake paintings of one master. At first the fakes are clumsy. The forger slips them into a pile of authentic works and hands the whole pile to an art dealer, who marks each piece real or fake and explains the verdicts. The forger returns to the studio and produces better fakes; meanwhile the dealer, seeing ever-better fakes, sharpens their own eye. Locked in this loop, each side forces the other to improve — until the dealer genuinely cannot tell the difference anymore. That loop is GAN training; now name its parts.

10.7.1 The Generator: From Noise Vector to Image

On one side, the generator $G$: it consumes random noise $z$ and manufactures data. Architecturally it resembles a CNN run in reverse — instead of shrinking an image down to features, it grows a tiny feature volume up into an image using transpose convolutions (also called fractionally strided convolutions). Where a normal convolution asks "summarize this patch into fewer numbers," a transposed convolution asks the opposite: "expand this summary over more pixels," increasing the size of the volume layer by layer.

A typical arrangement traces like this:

1. Take the incoming $128\times 1$ noise vector $z$.
2. Reshape (project) it into a small volume like $4\times 4\times 16$.
3. Stack transpose 2D convolution layers, each roughly doubling spatial size while reshaping channels: $4\times 4\times 16\to 8\times 8\to 16\times 16\to \cdots$
4. Stop when the output reaches, say, $128\times 128\times 3$ — a full RGB image.

All the convolution parameters are learned during optimization so that these images land close to the training distribution. Nothing about step 3 knows anything about faces or digits at initialization; every meaningful pattern in those kernels is discovered by training pressure from the adversary described next.

10.7.2 The Discriminator: A Binary Real-versus-Fake Classifier

On the other side, the discriminator $D$: a classifier, naturally built from ordinary CNN layers — exactly the kind of stack you have used for classification all course — whose job is to decide whether an image came from the real training set or from the generator. It learns to emit a high value — 1 — for genuine training images and 0 for anything the generator produces. Think of it as a binary classifier: real input → output near 1; generated input → output near 0. Its final score passes through a sigmoid so the number can be read as a probability of being real.

Note the elegant division of labor: the discriminator trains on labeled data for free — labels are just "real" or "fake," known by construction — so from its own perspective this is plain supervised binary classification. The subtlety is entirely on the generator's side.

10.7.3 The Adversarial Loop

Now the adversarial dance, run as a procedure:

1. Sample fake inputs. Draw a batch of noise vectors $z$ and generate images $G(z)$.
2. Assemble the training pool. Mix the generated batch with a batch of genuine training images.
3. Update the discriminator. Train $D$ on this mixed batch to say 1 for reals and 0 for fakes — an ordinary classification step.
4. Update the generator. Freeze $D$ for the moment and adjust $G$'s parameters so that the discriminator's verdict on fresh fakes moves toward "real."
5. Repeat. With each round the discriminator gets better at identifying genuine data, yet eventually fails to catch fakes — because the generator improves enough to fool it.

The generator's goal deserves emphasis: it watches the discriminator's verdicts and adjusts its own parameters so that its output moves toward the true training-data distribution — it wants to pull the discriminator into believing a generated image is actually real. Neither network learns an explicit distribution; the generator implicitly produces data that resembles the original so well the discriminator cannot tell the difference. At the equilibrium point, $D$ catches real data reliably yet fails on fakes — its answers sit at chance level — and only $G$ survives as the useful artifact.

Real-world: this same loop, with convolutional stacks on both sides, powers face generation, photo upsampling, and dataset augmentation pipelines — anywhere realistic synthetic imagery saves collection cost.

10.8 The Minimax Objective

10.8.1 Reading the Value Function Term by Term

This is arguably the single most important formula of the topic — the building block describing GAN behavior — and it looks unlike every loss seen before: not a pure minimization nor a pure maximization, but both at once. The value function:

$$\underset{G}{\min }\underset{D}{\max }V(D,G)={\mathbb{E}}_{x\sim p_{\text{data}}(x)}[\log D(x)]+{\mathbb{E}}_{z\sim p_z(z)}\left[\log \right(1-D(G(z))\left)\right]$$

Name every symbol. $z$ is the random noise vector — small-dimensional, say $128\times 1$, drawn from the noise prior $p_z$. $G$ is the generator, mapping that $128\times 1$ vector to, for example, a $128\times 128\times 3$ image. $x$ ranges over the real training data under the data distribution $p_{\text{data}}$, same $128\times 128\times 3$ shape. $D(\cdot )$ outputs a real number interpreted as "probability of being real." $V(D,G)$ is called a value function rather than a loss precisely because one player climbs it instead of descending it. The end product of all training is $G$: a network that takes $128\times 1$ vectors and creates semantically plausible $128\times 128\times 3$ images resembling the training set.

Maximize the whole expression with respect to $D$:

• For any genuine training datum $x$, a well-trained $D$ should output a value close to 1, making $\log D(x)$ close to 0 (the log of a number near one sits just under zero).
• For any fabricated $G(z)$, $D$ should output close to 0, making $1-D(G(z))$ close to 1 and so $\log (1-D(G(z)))$ close to 0.
• If $D$ is untrained and fails at this, the function value is a large negative number. Training $D$ raises the value toward 0. That ascent is the discriminator's education: all real images score near 1, all generated images near 0.

Minimize with respect to $G$: the generator wants to create $G(z)$ such that $D(G(z))$ is close to 1. Then $1-D(G(z))$ is close to 0 and its logarithm is strongly negative — pushing the minimization hard. Minimizing the $G$-part amounts to manufacturing data the discriminator scores as real, i.e., data very close to the original training distribution. Summing up: maximization over $D$ tunes the discriminator to certify training data as real and flag far-from-training data as fake; minimization over $G$ sharpens generation quality. Both happen simultaneously — a genuinely new kind of objective compared to everything earlier in the course.

10.8.2 Why GAN Training Is Unstable

Here is the honest bad news about GANs. Jointly optimizing a term that one player maximizes while the other minimizes creates training dynamics that are not very stable. Every generator step rewrites the very landscape the discriminator is climbing, and vice versa — so there is no fixed valley to descend. The loss frequently refuses to glide down asymptotically; instead it oscillates — drops, rises, drops again. This instability is the defining practical headache of GAN training, and upcoming sessions tackle it head-on by modifying the overall loss function and redesigning generator and discriminator architectures.

A useful way to picture it: ordinary deep learning is one hiker rolling downhill in fog. GAN training is two hikers on the same terrain, one told to reach the lowest point and the other the highest — each step either takes changes the ground under both feet.

10.8.3 The Original Training Loop (Pseudo-code)

The original algorithm interleaves the two players asymmetrically. One round proceeds as follows:

1. For $k$ steps, repeat: sample a mini-batch of $m$ noise vectors $z^{(1)},\dots ,z^{(m)}$ from the noise prior and a mini-batch of $m$ examples $x^{(1)},\dots ,x^{(m)}$ from the data-generating distribution; then update the discriminator by ascending its gradient:

$${\theta }_D\leftarrow {\theta }_D+\eta {\nabla }_{{\theta }_D}\frac{1}{m}\sum _{i=1}^m\left[\log D\right(x^{(i)})+\log (1-D\left(G\right(z^{(i)}\left)\right)\left)\right]$$

1. After those $k$ steps, sample a mini-batch of $M$ fresh noise samples and update the generator by descending stochastic gradient on its slice of the loss:

$${\theta }_G\leftarrow {\theta }_G-\eta {\nabla }_{{\theta }_G}\frac{1}{m}\sum _{i=1}^m\log (1-D(G\left(z^{(i)}\right)\left)\right)$$

Both updates are confirmed as written: each averages its per-sample terms over the mini-batch with the factor $1/m$, exactly the standard stochastic form of the original algorithm; the discriminator ascends because its terms enter the value function positively, while the generator descends because only the second term involves its parameters. Note the generator's sum contains no $\log D(x)$ piece — real images tell the generator nothing about how to improve.

Notes from the description: ${\theta }_D$ are the discriminator's parameters — ordinary CNN parameters; the noise comes from a random-number generator; nothing forbids adding momentum-based optimizers or similar extras. The count $k$ is a hyperparameter, often taken greater than 1: strengthen the discriminator first before letting the generator play. And throughout, the learning rates must be chosen carefully so the loss settles after some iterations instead of oscillating forever.

10.8.4 A Two-Player Game

This minimax structure is the same object game theorists study. Game theory was developed first in the economics community, originally to reason about demand-supply gaps, and the field's importance was recognized with a Nobel Prize in Economics. GANs import the two-player-game principle directly: the players are the discriminator and the generator, locked in the fight described above — $D$ trying to send $D(G(z))$ toward 0, $G$ trying to drag it toward 1. Where ordinary training seeks a minimum, this game seeks an equilibrium: a pair $(D^{\ast },G^{\ast })$ that neither player can improve on alone — the same notion of stable outcome economists use when neither firm can profit by changing strategy unilaterally.

10.8.5 The Non-Saturating Trick: Maximizing $\log D(G(z))$

People who trained early GANs noticed something intuitive: creating meaningful data from a tiny random vector is a harder task than discriminating real-looking images from fake ones. So at the start of training, $G$ is weak while $D$ is strong — and since $D$ dominates, $D(G(z))$ sits near 0. Now watch what that does to the generator's loss $\log (1-D(G(z)))$.

Write $s$ for the discriminator's raw score just before its final sigmoid, so $D(G(z))=\sigma (s)$. Differentiating both candidate objectives with respect to $s$:

$$\frac{\partial }{\partial s}\log (1-\sigma (s))=-\sigma (s),\frac{\partial }{\partial s}[-\log \sigma (s)]=\sigma (s)-1$$

With $\sigma (s)\approx 0$ early in training, the first derivative is about $0$ — the error signal reaching the generator is nearly flat, the same predicament as the vanishing gradient problem — while the second is about $-1$, full strength. Plug in numbers: at $D=0.05$ the old loss transmits a gradient scaled by $0.05$, the new one by $0.95$ — roughly nineteen times stronger. That is why, with the naive objective, the generator struggles to learn anything useful early on.

The fix people originally adopted: replace that sluggish term with

$$-\log D(G(z))$$

Early in training $D(G(z))$ is close to 0, so $-\log D(G(z))$ is large and its gradient is sharp — strong, usable information for updating $G$. Operationally: stop doing gradient descent of $G$ on $\log (1-D(G(z)))$; instead minimize $-\log D(G(z))$, equivalently perform gradient ascent to maximize just $\log D(G(z))$. Same equilibrium target, radically better early dynamics. This swap is the classic non-saturating trick from the original formulation — "non-saturating" because the new loss refuses to flatten out (saturate) when fakes are easily caught.

10.8.6 Early Results and Limitations of the Original GAN

Real-world: applied to face images and to MNIST digits, the original GAN produced reasonable generations, and nearest-neighbor checks against the training set show many samples looking quite close to some real example. Still, the honest assessment: promising for grayscale images of modest size, but performance degraded on larger color images. The variants covered in subsequent sessions do significantly better. Real-world: an interactive playground implemented in Keras lets you manipulate the training dynamics yourself and builds genuine insight into how the two losses interact.

10.9 Judging Generated Images I: The Inception Score

10.9.1 Why GANs Need External Judges

With every earlier generative model, judging output was easy in principle: you had learned an actual probability density, so you could pass a generated image back through it and see whether it received high probability — a low-probability score meant the sample sat far from the training distribution. GANs surrender that yardstick: no inferencing capability, no likelihood. So GAN-generated images require other ways of deciding whether they are reasonable. Two scores dominate practice: the Inception Score and the FID score.

10.9.2 Definition and the Two Properties It Rewards

The Inception Score (IS) for a GAN's outputs should be as high as possible. Mechanism: take a generated image and pass it through a large multi-class classifier trained on ImageNet — an Inception-style network with 1000 output classes, the architecture family introduced earlier in the course's CNN unit. The classifier produces class probabilities $p(y\mid x)$, where $x$ is the generated image and $y$ the predicted class label. Two properties are desired:

• Clarity / confidence. A generated image ought to resemble real images of some class, so the classifier should commit: one class probability high, the rest tiny. Equivalently, the entropy of $p(y\mid x)$ should be low. Recall that entropy, written $\mathcal{H}(p)=-{\sum }_{y}p(y)\log p(y)$, measures how spread out a distribution is: zero when all mass sits on one outcome, $\log K$ when it spreads evenly over $K$ outcomes. If a generated cat image yields a high cat-class probability and near-zero elsewhere, entropy is low, which is good. Contrast the bad case: softmax outputs like 0.25, 0.25, 0.25, 0.25 across four classes — summing to 1 as softmax guarantees — spread the mass evenly, and the entropy of that output is $\log 4$, its maximum, signaling a muddled, unclassifiable image.
• Variety / diversity. Across many generated samples, the class distribution should mirror the training data's spread. If training images contain cats, dogs, rats, elephants, the generated pool should contain a reasonably matching share of cats, dogs, rats, and elephants. That is captured by the marginal $p(y)$ — the average of $p(y\mid x)$ over all generated images — which should approach a uniform-like distribution, i.e., have high entropy.

Combining the two demands gives the definition:

$$\mathrm{IS}=\exp (\mathcal{H}(y)-\mathcal{H}(y\mid x))$$

where $\mathcal{H}(y)$ is the entropy of the marginal class distribution and $\mathcal{H}(y\mid x)$ the average conditional entropy of predictions given images. High IS arises exactly when variety is high ($\mathcal{H}(y)$ large — the generator is not regurgitating the same class repeatedly) and clarity is high ($\mathcal{H}(y\mid x)$ small — each image confidently belongs to some class). Notation note: many texts write the same score as the exponential of an expected KL divergence,

$$\mathrm{IS}=\exp \left(\frac{1}{N}\sum _{i=1}^N{D}_{\mathrm{KL}}\right(p(y\mid x_i)\Vert p(y)\left)\right),$$

which expands to the entropy form above once you write out the KL definition — two costumes, one quantity. A high IS certifies good quality in both variety and clarity — under one assumption: that the classifier's judgments track human perceptual similarity between generated images and the class concepts.

10.9.3 How One Image per Class Defeats the Score

But there is a hole in the score: a trivial case defeats it. Generate 1000 images, one for each of 1000 classes — one cat, one dog, one of everything else. Each image is confidently classified (low conditional entropy) and the marginal spreads evenly across classes (high marginal entropy). That set achieves a perfect-looking inception score, yet nothing about it says the images are diverse relative to the original training distribution or individually plausible beyond the classifier's verdict. A model that memorizes a single exemplar per class and replays it forever scores as well as a genuinely rich generator.

The IS does not genuinely measure diversity against real data — it never looks at real images at all. So the Inception Score alone is not enough; enter the FID score.

10.10 Judging Generated Images II: Fréchet Inception Distance

10.10.1 From Softmax Outputs to Deep Features

The Fréchet Inception Distance (FID) repairs the blind spot by comparing distributions in feature space rather than trusting class probabilities. It uses the Inception v3 network — the CNN classifier architecture from Google that concatenates features from multiple stages, delivers high classification quality, and has modest memory requirements.

Pass the training images through Inception v3 and collect the activations of its CNN layers; recall that normally a softmax submodule sits after those features to classify — for FID, discard the softmax entirely and keep only the CNN-derived feature values (in practice the deep "pooling" layer just before classification). From these features compute two statistics for the real dataset:

• $m$ — the mean vector of real features;
• $C$ — the covariance matrix of real features, which records how the feature dimensions co-vary across images.

Run the generated images through the same pipeline to get their mean $m_w$ and covariance $C_w$. The distance between these statistic pairs measures how far the generated feature cloud sits from the real one — closeness in this semantic feature space stands in for perceptual similarity.

10.10.2 The FID Formula and Its Behavior

$$\mathrm{FID}=\Vert m-m_w{\Vert }_2^2+\mathrm{Tr}\left(C+C_w-2(C{C}_w{)}^{1/2}\right)$$

This is the standard Fréchet distance between two multivariate Gaussians fitted to the two feature sets — exactly the form used for generative model evaluation. Here $\mathrm{Tr}(\cdot )$ is the trace, the sum of a matrix's diagonal entries, and $(C{C}_w{)}^{1/2}$ denotes the matrix square root — the symmetric matrix whose square equals $C{C}_w$. The first term penalizes shifted centers (different average content); the second penalizes differently shaped clouds (different spread and correlations).

Check the limiting case, as worked through below: if the covariances coincide, $C_w=C$, then $C+C_w=2C$ and the trace term collapses to nearly a zero matrix's trace — a small value. Combined with equal means ($m_w=m$), FID approaches 0. Conversely, whenever the means drift apart or the covariance structures differ substantially, FID grows large. Low FID is preferred for a good generative model — unlike the Inception Score, where high is good, remember the direction flips here.

10.10.3 Putting Both Scores Together

The working evaluation protocol: generate a batch of images while varying the random vector $z$; collect them; pass them through Inception v3; compute FID — which should be low — and also compute the Inception Score — which should be high. A good generative model shows both signatures simultaneously.

Why insist on joint reading? Evidence from stress tests: take a good generated image and add increasing levels of noise and blur. As disturbance grows, the Inception Score barely changes, but the FID climbs steeply — the two scores disagree, and FID is the one telling the truth. Other cases invert it: an image can post a high IS while also carrying a high FID, and a side-by-side look shows it is worse than an alternative with low FID. Images of genuinely good quality are reliably indicated by the pair: high inception score together with low FID. Together they certify that results have significant variety and resemble the training-image classes effectively.

10.10.4 Student Questions and Answers

A natural follow-up asked what to do when the numbers look bad.

Real-world: these two scores became the de facto report cards of generative vision research — papers quote FID curves epoch by epoch, and benchmark leaderboards rank models by low FID plus strong IS, keeping human eyeballs as the final arbiter above both.

Exam Guidance Summary

• Exam note: expect at least one component of assignment two to involve PixelCNN; the assignment lets you experiment with the VAE variants (beta-VAE, VQ-VAE, VQ-VAE-2) together with some GAN aspects.
• Exam note: the GAN minimax objective was flagged as the building block of the entire topic — know it cold, including the term-by-term reasoning for maximizing over $D$ and minimizing over $G$, and why an untrained discriminator leaves the value very negative.
• Exam note: be able to reconstruct the VQ-VAE three-piece loss (reconstruction + codebook + commitment, with $\beta$ weighting commitment) and explain which parameter set each stop-gradient protects: $\mathrm{sg}[z_e]$ freezes the encoder inside the codebook term; $\mathrm{sg}[e]$ freezes the codeword inside the commitment term.
• Exam note: know both evaluation metrics directionally — Inception Score high, FID low — and the failure story of each (IS defeated by one-image-per-class sets and by classifier overconfidence; FID rising under noise/blur while IS stays flat).
• Study advice: work through the recommended multi-part blog series covering GAN basics, GAN evaluation, and GAN variants; explore the Keras GAN playground for intuition about training dynamics.
• Roadmap: multiple subsequent sessions continue with GAN variants — including the fully convolutional DCGAN — and with fixes for training instability, so treat this session's objective and pseudo-code as foundation material for what follows.

A compact revision order that matches this session's structure: first rebuild the VQ-VAE loss and its stop gradients from memory; then write out the minimax value function and narrate each term's behavior under trained, untrained, and fooled discriminators; finally rehearse both evaluation scores with their directions and failure cases.

Key Industry Applications

• Real-world: VQ-VAE powers speech modeling — male-to-female and speaker-to-speaker voice conversion, general voice style transfer — using WaveNet decoders from autoregressive audio modeling.
• Real-world: VQ-VAE handles ImageNet-scale image reconstruction successfully, and extends naturally to one-dimensional signal compression and generation.
• Real-world: VQ-VAE-2's clarity competes with GAN-generated imagery; its hierarchical coarse-to-fine scheme mirrors human painting workflows (outlines first, details conditioned on outlines).
• Real-world: GANs progressed from grayscale toys in 2014 to very high-resolution, large images from 2021 onward; DCGAN established the fully convolutional recipe that later variants built on.
• Real-world: Inception Score and FID rely on Inception-network classifiers trained on ImageNet — the same Google-origin architectures reused as perceptual measuring instruments, quoted epoch by epoch in research papers and benchmark leaderboards.
• Real-world: GAN training formalizes as a two-player game — mathematics pioneered in economics (demand-supply analysis) and recognized with a Nobel Prize in Economics.
• Real-world: PixelCNN serves both as an image prior and as the codebook-index prior enabling VQ-VAE sampling; PixelSNAIL upgrades that role in the hierarchical variant.

Together these threads show one pattern: the same small set of architectures keeps reappearing in new roles — CNNs as encoders, decoders, discriminators, and judges; autoregressive models as image generators and then as priors over discrete codes. Recognizing which role a network plays in a pipeline is most of the work in reading modern generative-system papers.