Improving GAN Training: Optimal Discriminators, DCGAN, and the Wasserstein Distance

11.1 Where GANs Stand Among Generative Models

The roadmap from here is short to state. The study of GAN training continues in the coming meetings. Then the course turns to diffusion models together with energy-based models. After that come applications of these ideas in natural language processing and vision, and a full recap meeting closes the taught material. A dedicated problem-solving webinar drills exam-style questions for the finals. Keep that arc in mind, because everything built today (a better distance measure and a better GAN) is a direct fix for the failures listed below.

11.1.1 Explicit Versus Implicit Generative Modeling

Every generative model met so far except the GAN is an explicit model trained by maximizing likelihood. Explicit means you can write down the probability density of your data under the model — an actual formula $p(x)$ that you can evaluate at any point.

• Autoregressive (AR) models factor the joint probability into a product of conditionals: $p(x)={\prod }_{t=1}^{T}p(x_t\mid x_1,\dots ,x_{t-1})$, where each factor $p(x_t\mid x_{<t})$ is a network output over the $t$-th element given the earlier ones. They give an explicit and accurate distribution.
• Flow models invert a stack of easy transformations. They also give an explicit, accurate distribution: start from a simple base density and apply an invertible map with a known Jacobian determinant, so the density of the output can be computed exactly.
• Variational autoencoders (VAEs) maximize the data likelihood only in approximation, because the KL divergence term between the encoder and decoder distributions is intractable. The model still works, but it optimizes a bound instead of the true likelihood.

A useful everyday picture: an explicit model is like a bakery that publishes its exact recipe book — for any proposed cake, it can look up precisely how likely that cake is under its recipes. An implicit model is a forger with no recipe book at all: it can only produce convincing cakes, one after another. Ask the forger "how probable is this specific cake?" and it has nothing to say. That is exactly the trade the GAN makes.

A GAN takes the implicit route. No formula for the data density is ever written down or approximated. Instead two networks play a game:

1. A generator $G$ maps a small random vector $z$ (say 100 numbers) to a fake sample $G(z)$.
2. A discriminator $D$ looks at a sample and outputs how real it believes it is. Its goal: high output on true training data, near zero on generated data.
3. The generator's goal is the opposite: pull the discriminator's output up on fakes.

This tug-of-war is written as a minimax optimization: one player minimizes, the other maximizes, over the same objective function. Because both maximization and minimization are involved at the same time, training such a network is hard in principle. Train one yourself and you will often watch the loss oscillate; it does not settle smoothly down to a minimum the way ordinary supervised losses do. That instability is not a bug in your code. It is baked into the game.

11.1.2 Scoring Generated Samples: Inception Score and FID

Two properties define a good generator. First, its samples should be confidently recognizable: the discriminator should classify them into the real bucket without hesitation. Second, they should be diverse, matching the variety seen in the training data. The Inception Score (IS) packs both demands into one number. In plain words, the verbal form was: take e raised to the power of two entropies — the entropy of the label spread of generated data, minus the entropy of the per-sample label predictions. Reconciled against the standard definition, the formula is:

$$\mathrm{IS}=\exp (H(p(y))-{\mathbb{E}}_{x\sim p_g}[H(p(y\mid x))\left]\right)=\exp \left({\mathbb{E}}_{x\sim p_g}\right[\mathrm{KL}(p(y\mid x)\Vert p(y))\left]\right)$$

Here $x$ is a generated sample drawn from the generator's distribution $p_g$, $y$ is the class label predicted by a separate pre-trained classifier (historically the Inception network, which gives the score its name), $p(y)$ is the marginal label distribution averaged over all generated samples, $p(y\mid x)$ is the label distribution for one specific sample, and $H(\cdot )$ is entropy, $H(p)=-{\sum }_k{p}_k\log p_k$, where the sum runs over all $K$ classes.

The two entropy terms measure the two demands separately:

• Low conditional entropy $H(p(y\mid x))$: each single image gets a confident, peaked class prediction — quality of individual samples.
• High marginal entropy $H(p(y))$: across the whole collection, all classes appear roughly equally often — diversity of the set.

Why are the two forms equal? Expand the KL term and average it over samples. For each $x$,

$$\begin{array}{rl}{\mathbb{E}}_x[\mathrm{KL}(p(y\mid x)\Vert p(y))]& ={\mathbb{E}}_x\left[\sum _{k=1}^{K}p(k\mid x)\log \frac{p(k\mid x)}{p(k)}\right]\\ & =-{\mathbb{E}}_x[H(p(y\mid x))]-\sum _{k=1}^K\underset{=p(k)}{\underbrace{\left({\mathbb{E}}_x\right[p(k\mid x)\left]\right)}}\log p(k)\\ & =H(p(y))-{\mathbb{E}}_x[H(p(y\mid x))].\end{array}$$

The first move is the definition of KL divergence; the second swaps the order of averaging so that ${\mathbb{E}}_x[p(k\mid x)]$ collapses exactly to the marginal $p(k)$, whose negative sum is again an entropy. This confirms the equivalence claimed in the lecture's verbal description.

The catch: a high IS does not guarantee images that look good to a human eye. Perceptual quality can lag behind the score. The FID score was formulated to close that gap: instead of comparing class labels, it compares feature statistics (means and covariances of activations) of real and generated images inside a pre-trained network, so it rewards images whose internal statistics genuinely match the training set. A low FID combined with a high IS tracks human judgment of quality much better than either number alone.

Real-world use: IS and FID remain the standard reporting metrics for image generators, so reading any generative-model paper means reading these two numbers. Every architecture comparison you will meet later in this course (DCGAN baselines, WGAN improvements) is scored on exactly this axis.

11.2 What Makes a GAN Fast — and Where It Struggles

11.2.1 Generation Is One Forward Pass

Sampling from a GAN is fast because generation is just arithmetic through a deep network. Change the entries of a small random vector, say 100 by 1, and pass it through the trained generator. Out comes new data, possibly shaped like an RGB image of size 224 by 224 by 3 if that matches your training set. No iterative procedure, no posterior inference, no Markov chain. Just one forward pass. This is why GANs earned a reputation for fast sampling compared with every likelihood-based alternative.

It helps to see the contrast on dimensions that matter when you actually train these systems:

When to pick which: if you need fast bulk generation of perceptual media, the GAN route wins; if you need to score or rank individual points by probability, only the explicit side can do it.

The GAN objective also optimizes directly for what you actually care about: whether samples look real enough to fool a judge, rather than whether some surrogate log-likelihood went up. That alignment between goal and loss is a genuinely attractive property.

11.2.2 Known Weak Points

Everything in the rest of this document attacks these weak points, first with engineering (DCGAN, improved techniques) and then with mathematics (the Wasserstein distance).

Real-world placement: these trade-offs decided industrial practice. When a product needs millions of images quickly — game asset prototyping, data augmentation for rare defect classes in manufacturing inspection — the one-forward-pass sampler was worth the training pain, and teams accepted that they could not rank individual outputs by likelihood.

11.3 Theory of the Optimal Discriminator

Before improving anything, pin down what a perfect discriminator would even look like. To find it, build a functional — a number computed from a whole function — that captures the discriminator's total performance tied to the GAN's goal, then find which function maximizes it. The basics come straight from a machine learning prerequisite: optimal thresholds and Bayes-style decisions.

11.3.1 The Discriminator's Objective Functional

Let $x$ denote a training sample drawn from the true data distribution $p_{data}$. Let $z$ be the small random input vector fed to the generator, drawn from a simple prior $p_z$ such as uniform noise. Let $G(z)$ be the generated fake and $D(x)\in [0,1]$ the discriminator's realness score for any input. The value functional for the discriminator is:

To work with this, recall from a probability course how an expectation becomes an integral: for a random variable with density $p(x)$, the expected value is $\int xp(x)dx$ over the whole range. Applying that here:

$$V(D)=\int p_{data}(x)\log D(x)dx+\int p_z(z)\log (1-D(G(z)))dz$$

Now one clean substitution. Generated values $G(z)$ follow the generator's own distribution $p_g$. Replace the variable $z$ by $x$ distributed according to $p_g(x)$; this is just pushing the noise prior through $G$:

$$V(D)=\int \left[p_{data}(x)\log D(x)+p_g(x)\log (1-D(x))\right]dx$$

All we did was replace $G(z)$ with $x$ and $p_z(z)$ with $p_g(x)$, the generator's probability distribution.

11.3.2 Deriving the Optimal Discriminator Step by Step

Look inside the integral at a single point $x$. Make three renamings: call $p_{data}(x)$ simply $a$, call $p_g(x)$ simply $b$, and call $D(x)$ simply $y$. The integrand becomes:

$$f(y)=a\log y+b\log (1-y)$$

Maximizing the integral means maximizing this expression at every $x$ independently, since $D$ can be chosen freely at each point. Why can we treat each point separately? Because the integral just adds up the integrand's contributions point by point — no choice of $D$ at one location changes the payoff at another. So the best global function is the pointwise best value everywhere.

The optimality condition for a smooth one-dimensional function is that its derivative equals zero. Differentiate term by term ($\frac{d}{dy}\log y=\frac{1}{y}$ and, by the chain rule, $\frac{d}{dy}\log (1-y)=\frac{-1}{1-y}$):

$$\frac{d}{dy}\left[a\log y+b\log (1-y)\right]=\frac{a}{y}-\frac{b}{1-y}$$

Setting that derivative to zero and solving gives the optimal value of $y$. Here is every line of algebra:

$$\begin{array}{rlrl}\frac{a}{y}-\frac{b}{1-y}& =0& & \text{(set the derivative to zero)}\\ \frac{a}{y}& =\frac{b}{1-y}& & \text{(move the second term across)}\\ a(1-y)& =by& & \text{(cross-multiply by }y(1-y)\text{, valid since }0<y<1\text{)}\\ a-ay& =by& & \text{(distribute }a\text{)}\\ a& =y(a+b)& & \text{(collect the }y\text{ terms on one side)}\\ y^{\star }& =\frac{a}{a+b}.& & \text{(divide by }a+b\text{)}\end{array}$$

This critical point really is a maximum, not a minimum: differentiate once more,

$$\frac{d^2}{d{y}^2}\left[a\log y+b\log (1-y)\right]=-\frac{a}{y^2}-\frac{b}{(1-y{)}^2},$$

and both terms are strictly negative for every $a>0$, $b>0$ and every $y\in (0,1)$. A negative second derivative everywhere means the function is concave, so the single critical point is the global maximum. As a boundary sanity check, $f(y)\to -\infty$ as $y\to 0^{+}$ (when $a>0$) and as $y\to 1^{-}$ (when $b>0$), confirming the maximum lives strictly inside the interval.

Undoing the renaming ($y=D(x)$, $a=p_{data}(x)$, $b=p_g(x)$) yields the central result:

The best possible discriminator is a ratio of densities. It needs no training tricks; it is exactly what the game forces.

11.3.3 Geometric Meaning: The 0.5 Threshold

Recall from the ML toolkit the picture of two overlapping bell curves where the optimal classification threshold sits at the crossing point of the two density curves, giving minimum error. The formula says precisely that. Picture the horizontal axis as pixel space and the vertical axis as density. Wherever $p_{data}(x)>p_g(x)$, the ratio exceeds one half, so the optimal discriminator calls the point real. Wherever $p_g$ wins, it calls the point fake. Exactly at the crossings the ratio equals one half — the decision flips there, which is why those crossing points are the minimal-error threshold locations from the prerequisite course.

Special case worth memorizing: if the generator has fully learned the data distribution, then $p_g=p_{data}$ everywhere and:

$$D^{\star }(x)=\frac{p_{data}(x)}{2p_{data}(x)}=\frac{1}{2}\text{for all }x$$

A perfect generator leaves the discriminator unable to tell anything apart, stuck at one half on every input. Output above one half goes to the data class; below, to the generated class.

11.3.4 The Best Achievable Generator Loss

Plug the optimal discriminator back into the value function. Writing $m=\frac{p_{data}+p_g}{2}$ for the midpoint distribution and using $D^{\star }=\frac{p_{data}}{p_{data}+p_g}=\frac{1}{2}\cdot \frac{p_{data}}{m}$, the generator's cost becomes:

$$\begin{array}{rl}C(G)& ={\mathbb{E}}_{x\sim p_{data}}[\log D^{\star }(x)]+{\mathbb{E}}_{x\sim p_g}[\log (1-D^{\star }(x))]\\ & ={\mathbb{E}}_{x\sim p_{data}}\left[\log \frac{1}{2}+\log \frac{p_{data}}{m}\right]+{\mathbb{E}}_{x\sim p_g}\left[\log \frac{1}{2}+\log \frac{p_g}{m}\right]\\ & =-\log 2-\log 2+\mathrm{KL}\left(p_{data}\Vert m\right)+\mathrm{KL}\left(p_g\Vert m\right).\end{array}$$

The first line substitutes the optimal discriminator; the second splits each logarithm using $\log \frac{p}{2m}=\log \frac{1}{2}+\log \frac{p}{m}$; the third recognizes each remaining expectation as a KL divergence by definition. Collecting constants gives the form stated in the lecture:

Both KL terms are non-negative and vanish only when $p_g$ coincides with $p_{data}$. So the floor of the generator's cost is:

$$C(G)\ge -\log 4,\text{with equality iff }{p}_g=p_{data}$$

For contrast, consider the non-GAN setting: training a plain maximum-likelihood classifier. That objective also bottoms out when the model distribution equals the data distribution. The idealized minimum there lines up with the same minus-log-four figure. In practice real runs stay above the floor because the generator never matches the data perfectly, leaving a gap between the two distributions. Knowing the floor calibrates how you read a training curve: a generator loss hovering well above $-1.386$ means the two distributions still differ measurably.

11.4 Comparing Distributions: Forward KL, Reverse KL, Mode Collapse, Mode Covering

Why spend time on distance functions? Because the plan is to build an improved GAN variant whose objective uses a distance between two distributions, chosen deliberately instead of inherited blindly. Understanding what each distance rewards and punishes tells you which failure mode your GAN will develop.

11.4.1 A Toy Mixture of Two Gaussians

Take a one-dimensional example simple enough to draw. The true data distribution is a mixture of two Gaussians with nearly equal standard deviations, differing only in their means, mixed half and half:

$$p(x)=0.5\mathcal{N}(x;{\mu }_1,{\sigma }^2)+0.5\mathcal{N}(x;{\mu }_2,{\sigma }^2)$$

Here $\mathcal{N}(x;\mu ,{\sigma }^2)$ is the Gaussian (bell-curve) density centered at mean $\mu$ with variance ${\sigma }^2$, and the two coefficients 0.5 are mixture weights — shares of total probability assigned to each bump, summing to one. So the training data piles up in two bumps with a nearly empty valley between them. Real datasets are usually multimodal like this, and the picture grows far more complex when the data has many dimensions instead of one.

Visual intuition: plot density on the vertical axis against $x$ on the horizontal axis. You see two peaks of almost equal height at ${\mu }_1$ and ${\mu }_2$, separated by a valley that dips nearly to zero between them. The single number that matters later is each peak's share of total area — change the share and the peak heights move even though the centers stay fixed.

11.4.2 Forward KL Gives Mode Covering

The forward KL divergence from $p$ to a candidate model $q$ is:

Minimizing it forces $q$ to put mass wherever $p$ has mass, including the low-lying tails. The reason sits inside the logarithm: wherever $p(x)>0$ but $q(x)$ approaches zero, the ratio $\frac{p(x)}{q(x)}$ explodes toward infinity, so leaving out any region the data touches costs an unbounded amount. For the two-bump toy problem, the minimizing $q$ spreads across both bumps and the valley between them. Now generate samples from that $q$: many land on the modes, but plenty land in regions where real data is rare. Those samples are very unlikely under the true data distribution — they look unlike anything in the training set. Variety survives; perceptual quality suffers. This failure pattern is called mode covering: the generator covers every mode but pays for it with implausible samples filling the empty middle.

Real-world connection: this same pattern is an indication of hallucination. The word hallucination is used mostly for text generators, but an image generator emitting samples no one recognizes as meaningful data fails the same way, and over-wide coverage is one cause.

11.4.3 Reverse KL Gives Mode Collapse

Flip the argument order:

Now $q$ is terrified of placing mass where $p$ is near zero, because that makes the log ratio explode. The cheapest escape is to hug a single bump of $p$ tightly. The fitted $q$ collapses onto one mode. Samples look sharp and highly realistic — perceptual quality stays high — but every sample has the same flavor. Diversity dies. This is the famous mode collapse problem of GANs: the generator collapses to one of the modes of the training distribution. The complementary vocabulary: mode seeking behavior seeks one mode, and collapsing is what kills variety.

So remember this pairing, because the improved GAN later today is constructed directly around escaping it: forward KL keeps variety and loses quality; reverse KL keeps quality and loses variety.

Pick forward KL when missing real regions is the worst outcome; pick reverse KL when emitting garbage outside the data is the worst outcome. GANs inherit pressures from both, as the next subsection shows.

11.4.4 Jensen–Shannon Divergence: An Average With the Same Traps

The Jensen–Shannon divergence (JSD) is nothing more than an average of the forward KL and the reverse KL, taken against a midpoint distribution. Both failure pressures survive inside the average: one side pulls toward covering, the other toward collapsing. The original GAN objective, it turns out, ties to JSD — which is why the saturation problems developed below matter so much. The formal definition and its geometric quirks get their own treatment in Section 11.7.

11.4.5 Classroom Question: Why Did the Peak Move?

Two versions of the mixture picture were shown side by side. In the second, the blended curve's peak had drifted toward the left bump instead of sitting midway.

Several students made the same "higher mean" guess, so here is the correction once more, stated directly: peak position encodes mixture weights, and mixture weights are exactly what a collapsing generator lets degenerate.

11.5 DCGAN: Deep Convolutional GAN

The first upgrade is architectural. The deep convolutional GAN (DCGAN) is a GAN tuned specifically for image generation, built on CNN machinery. Recall the convolutional autoencoder assignment: there the goal was good reconstruction. Here the goal differs — generating fresh images from noise — but the building blocks overlap heavily.

11.5.1 Generator Architecture Walkthrough

The generator starts from a 100-dimensional random vector and ends at a 64 by 64 by 3 RGB image, matching a training set of 64 by 64 color images. Follow the tensor shapes:

Read the table as a story: the spatial size doubles at every step while the channel count halves — increasing spatial resolution, reducing depth. The first stage is a plain multi-layer perceptron with 100 inputs whose outputs are reshaped into a 4 by 4 grid with 1024 channels; those channels behave like feature maps. From there on, every layer is a fractionally strided convolution, also called a transposed convolution (use Conv 2D transpose). This is the operator designed to grow spatial size, going 4 by 4 up to 8 by 8, then onward to 64 by 64.

How does one layer double the grid? A transposed convolution spreads each input cell over a wider patch before combining, which grows the output side. With kernel size $k=4$, stride $s=2$, and padding $p=1$, the output side length is $(n_{\text{in}}-1)\cdot s-2p+k$. Check it on the first upsample: $(4-1)\cdot 2-2\cdot 1+4=6-2+4=8$ — exactly the $8\times 8$ in the table. Run the same arithmetic on 8 and you get 16, then 32, then 64. The doubling in the table is this formula applied four times.

Once training finishes, throw the discriminator away. Generating data is now just rolling the random vector again and again and passing each roll through the generator module. Sampling is cheap, entirely neural-network based, and fast — change the 100 numbers, get a brand-new image.

11.5.2 Discriminator Design and Training Rules

In principle the discriminator could be any standard CNN classifier doing two-class discrimination: real versus generated. In practice, vanilla choices fail under the adversarial loss, so the DCGAN recipe applies engineering modifications. These are optimized settings, and the system is famously very sensitive: change the architecture carelessly and training breaks. Obey the recipe and reasonable training follows for many datasets of smallish images.

Good samples have been produced this way on datasets containing around 3 million images, provided the images stay small.

11.5.3 Results: Faces and Interpolations

Trained on face collections, the generator produces convincing synthetic humans purely by varying the input noise vector. Look at the variety: different skin tones, different genders, people wearing glasses and people without, varying ages, lots of hair and bald heads, young faces and older ones. Every portrait is fabricated; none corresponds to a photographed person.

The model also behaves well in high-dimensional space: interpolations come out smooth. Feed two views of the same scene — two viewpoints of one environment — and the network can synthesize intermediate views, nice-looking images that were never captured. That interpolation ability doubles as evidence that the generator learned a structured representation, not a lookup table: between any two learned points there is a sensible path, which only happens when nearby latent codes map to visually nearby outputs.

Limits repeat from before: chase larger images while keeping clarity and performance decays.

11.5.4 Vector Arithmetic on Latent Semantics

The most beautiful DCGAN discovery: the latent space supports vector arithmetic. Pass many random vectors $z$ through the generator, collect the outputs, and visually group them by attribute — say three smiling women generated from $z_1^s,z_2^s,z_3^s$, alongside neutral women and neutral men.

The glasses trick repeats the pattern: subtract men-without-glasses vectors from men-with-glasses vectors to isolate "glassesness," then add that direction to a woman's vector and get a woman wearing glasses.

You are playing with the semantics of the data inside the input vector of the generator. Attributes became addressable coordinates.

Real-world connection: many photo editing applications improved significantly thanks to exactly this kind of manipulation, and these synthesized faces double as fake-but-realistic training or display imagery. The same idea — find a direction in a learned latent space that encodes one human-meaningful factor — now powers editable generative tools far beyond faces.

11.5.5 Strengths, Limits, Takeaways

Summary judgment on DCGAN. Incredible samples for a generative model of its era, especially given that sampling costs one forward pass. Not so good for very large images. Works well when the specific architectural details above are respected; without those specifics, expect trouble. Interpolation and representation learning come free. The open problems: unstable training, brittle architecture hyperparameters — an awful lot of tweaking. Those two problems motivate the principled fixes coming next.

11.6 Improved Techniques for Better GAN Training

One influential paper collected a toolbox of generic stabilization recipes for GAN training. Later work moved past parts of it, but the paper mattered enormously, and the recipes apply generally whenever GAN training misbehaves. One idea each, briefly.

11.6.1 Feature Matching

The discriminator is a stack of layers, and its intermediate layers already extract features from whatever passes through. Feature matching exploits them. Compute the average feature vector of real training data at some intermediate layer. Compute the average feature vector of generated data at the same layer. Then minimize the distance between the two averages, in addition to the usual adversarial objective:

If generated images truly resemble training images, their internal features should match too, not just their surface success at fooling the judge. This added pressure often stabilizes training, because it gives the generator a target that stays informative even when the discriminator itself becomes easy to fool.

11.6.2 Minibatch Discrimination

Mode collapse is the enemy here. Real batches vary richly; collapsed generated batches do not. So give the discriminator a side channel. For each sample, compute a similarity summary against the other members of its batch — call it $o(x)$, a vector describing how sample $x$ relates to its batchmates. Feed the discriminator both the usual CNN features and this similarity vector. A healthy batch shows a spread of moderate similarities; a collapsed generated batch is abnormally self-similar, and that deviation from healthy batch statistics is exactly what the discriminator sees through $o(x)$. The generator then gets penalized for producing batches that all look alike. Net effect: a detector aimed squarely at mode collapse.

The textbook framing says the same thing in one line: minibatch discrimination lets the discriminator send a signal encouraging the generator to include a similar amount of variation as the original dataset.

11.6.3 Historical Averaging

Training also benefits if parameters refuse to wander wildly. Historical averaging adds a penalty keeping the current parameter vector ${\theta }_t$ close to the running average of all previous parameter vectors:

The new theta should not stray far from the average of every theta observed in earlier iterations. This damps the oscillation typical of minimax games: if the two players keep trading positions back and forth around an equilibrium, the average barely moves, so any wild swing pays a penalty.

11.6.4 One-Sided Label Smoothing

When computing cross-entropy, do not demand that real data produce exactly 1 at the discriminator. Use 0.9 instead for real samples only — one-sided smoothing. Why? A discriminator pushed toward extreme confidence on reals becomes overtrained, and an overconfident judge hands the generator useless gradients. Softening the target keeps the judge humble and trains the generator better.

A tiny numeric picture of the effect: suppose the discriminator currently outputs 0.99 on a real image. With target 1.0 its cross-entropy is $-\log (0.99)\approx 0.01$ nats — almost no learning signal remains, and the judge keeps sharpening toward saturation. With smoothed target 0.9 the loss becomes $-[0.9\log (0.99)+0.1\log (0.01)]\approx 0.009+0.461=0.47$ nats, a healthy gradient that keeps the judge's slope alive. Only the real label is softened ("one-sided"); fake targets stay at 0, because softening those would encourage the discriminator to tolerate obvious fakes.

11.6.5 Virtual Batch Normalization

Ordinary batch normalization normalizes using the current mini-batch, letting samples within a batch co-adapt. Virtual batch normalization changes the reference point: fix one reference batch of 128 real images, sampled once from the training set and kept aside. Normalize each new example using statistics computed over that example plus the reference batch. Layer inputs stay stationary across steps without within-batch co-adaptation — helpful mainly in the generator. The trade-off is speed: every forward pass now drags 128 extra images along for the statistics, so it is reserved for situations where co-adaptation actively hurts.

11.6.6 Semi-Supervised GAN

If labels exist for some training points, use them. Widen the discriminator's output from binary real-or-fake to $k+1$ outputs: the first $k$ correspond to the known classes, the extra one to "fake." The loss splits:

The requirement is access to at least some labeled training data. The payoff runs both ways: the classifier learns from unlabeled data through the GAN signal, and the generator receives richer feedback because the judge now understands class structure.

11.6.7 Projected GANs and Pre-Trained Feature Spaces

Modern twist: run the adversarial comparison not on raw pixels but inside a pre-trained network's feature space. Suppose $f_L$ denotes frozen layers of a pre-trained backbone — think Inception V3, or ResNet variants. Project both real images $x$ and generated images $G(z)$ through those layers. Then apply the GAN principle to the projections: discriminators operate on the frozen features at multiple scales, and their adversarial losses are combined into one objective summed over the available layers $L$. Payoffs: improved image quality and reduced training time, because the heavy lifting of representation learning is borrowed rather than relearned.

This idea rides a broader truth of modern deep learning: rarely do you have the data or the compute budget to start from scratch. Pre-trained models exist in abundance, and using one adapted to your task beats building fresh — pre-training and transfer learning are the practical workhorses of industrial deep learning.

Real-world connection: projected-GAN thinking is the same muscle as industry transfer learning — freeze a strong backbone, train only what your task adds.

11.7 Distances Between Data Manifolds

11.7.1 The Data Manifold Picture

Simplify to two dimensions to see the geometry. Plot the training data after projecting it down — or grab any two dimensions and scatter the points. The cloud of points traces a data manifold: the low-dimensional surface on which real data concentrates. A manifold, in this context, means exactly that — a curved lower-dimensional sheet inside the high-dimensional pixel space, the way a sheet of paper crumpled in your fist is still a 2D surface living in 3D space. Now overlay the generator's outputs, sampled across many noise inputs. They form a second cloud. Training a generator means closing the gap between the two clouds — pulling every generated point closer to the manifold of real data. A distance function is precisely the measuring stick that defines "closer" and drives that movement.

Visual intuition: horizontal plane = the two projected dimensions; height (or shading) = density. The real data forms one shaded ridge; generated samples form a second ridge somewhere else on the plain. Early in training the ridges barely touch; each step of good training bends and slides the generated ridge toward the real one. Remember the caveat: the real fight happens in a huge space whose dimensionality is pixels times channels; the sketch is a cartoon of that high-dimensional reality.

11.7.2 JS Divergence: Symmetric but Saturating

Name the two parties: $p_r$, the real data distribution, and $p_g$, the generator's distribution. The goal is a distance whose reduction drags the generated cloud onto the real one. Start from KL's shortcomings. KL is asymmetric: traveling from $p_r$ to $p_g$ is not the same journey as returning. The Jensen–Shannon divergence fixes the asymmetry by averaging two KL terms against the midpoint $M=\frac{P+Q}{2}$:

Its behavior: JSD equals zero exactly when the two distributions coincide, rises while they partially overlap, and then saturates at $\log 2$ once their supports stop overlapping entirely. Why $\log 2$? Fill in the algebra for the fully disjoint case. Where only $P$ has mass, $Q$ contributes nothing to the midpoint, so there $M=P/2$; likewise $M=Q/2$ on $Q$'s own support. Then:

$$\begin{array}{rl}\mathrm{KL}(P\Vert M)& ={\int }_{\text{supp}(P)}P(x)\log \frac{P(x)}{P(x)/2}dx={\int }_{\text{supp}(P)}P(x)\log 2dx=\log 2,\end{array}$$

and identically $\mathrm{KL}(Q\Vert M)=\log 2$. Half of each term sums back to $\log 2$. That saturation is fatal for GAN training. Two disjoint distributions separated by a mile and two disjoint distributions separated by a continent both report $\log 2$. The number carries no gradient information about how far apart things are.

And early training is precisely the disjoint regime. The generator starts terrible, so its distribution sits far from the data. The judge learns quickly while the generator lags — and JSD sits pinned at its ceiling, whispering nothing. Only in the narrow range where the distributions overlap does JSD behave usefully. Worse, at the moment of first contact the value jumps discontinuously from $\log 2$ to smaller values, a kink that makes optimization awkward. Picture the curve: distance between clouds along the horizontal axis, JSD value rising vertically — flat ceiling plateau, then a cliff down at first overlap. A better yardstick is needed.

Real-world placement: this analysis is why practitioners watch sample quality stall early in vanilla GAN training. The objective stopped giving directions while the generator was still hopeless — the failure mode Section 11.8 is built to remove.

11.8 The Wasserstein Distance

11.8.1 Earth Mover Intuition

Enter the Wasserstein distance, also called the Earth Mover's distance. The lecture's picture: two piles of sand with different shapes. Your job: move sand from one pile until it exactly resembles the other. The Earth Mover's distance is the energy spent in that reshaping — equivalently, the minimum amount of sand that must be lifted and carried to turn one pile into the other. Small effort means the piles were similar to begin with; huge effort means they were far apart. The metaphor scales down beautifully to probability distributions, which are just piles of unit volume split across locations.

Where the analogy maps cleanly: sand = probability mass, positions = values of $x$, carrying work = mass times distance moved, and "the cheapest overall moving plan" = the minimization in the formula below. Where it breaks: real sand can be dumped anywhere, while a transport plan must respect exact column-by-column bookkeeping — every unit taken from somewhere must land somewhere specific.

11.8.2 Shovel-by-Shovel Worked Example

Make the piles discrete. Draw histograms where each block is one shovel-load of sand, laid out over five adjacent columns. Pile one has column heights $[0,3,0,4,1]$; pile two has $[2,1,2,1,2]$. Both piles contain the same total volume — eight shovels — matching the rule that probabilities sum to one.

Many plans other than the optimal one also transform pile one into pile two — route different shovels along different boundaries and the piles still match at the end — but those plans cost more. The Wasserstein distance is defined by the best plan: the minimum-cost way to reshape one distribution into the other.

Note the constraint that makes this a probability tool rather than a general one: both piles must hold equal total volume, which for distributions means both sum to one.

11.8.3 Formal Definition Through Optimal Transport

Write it mathematically. Let $\Pi (p_r,p_g)$ be the family of joint distributions $\gamma$ whose marginals are $p_r$ and $p_g$ — the Cartesian product coupling of the two laws. Each $\gamma$ is one transportation plan specifying how much mass travels from point $x$ to point $y$.

Why not just compute this directly? Count the work. Ten support points on one side and ten on the other means reasoning over pairings of size order $n^2$, about 100 pairs. And every one of exponentially many plans must be evaluated to certify the infimum. With continuous, high-dimensional distributions the enumeration explodes. Evaluating the Wasserstein distance exactly is an optimal transport problem, and it is intractable head-on — solvable in principle as a linear program (minimize the shipping cost subject to the supply-and-demand constraints), but that remains expensive at scale. Nothing comes free in machine learning: an honest, well-behaved distance turns out to be expensive to compute exactly. The blog article recommended alongside this topic develops the concept further, discrete and continuous alike, and is well worth the read.

11.8.4 Non-Overlap Stress Test

Line the contenders up on the worst case. Let $p$ be a lump concentrated at the origin and $q$ the identical lump shifted rightward by $\theta$, so the two fail to touch for any $\theta \ne 0$.

Read the rows. KL explodes to infinity the instant the supports separate — the lump sits where the shifted lump assigns zero density — and infinities are miserable to optimize. JS freezes at $\log 2$ regardless of separation: useless exactly when guidance matters most, and nondifferentiable at contact, jumping between 0 and $\log 2$. The Wasserstein distance returns $|\theta |$ — literally the distance the lump traveled. At zero shift everything agrees at zero; as shift grows, only W keeps growing sensibly, always proportional to the actual gap. Quick sanity checks: every row is non-negative as a distance must be; shrinking $\theta$ continuously shrinks only W smoothly to zero; and W is symmetric under swapping the two lumps, matching $|\theta |$. So minimizing W drags the generator toward the data from any starting distance, which is exactly the property a GAN loss needs. That smooth, informative behavior is the goodness of the earth mover's distance.

Real-world placement: optimal transport now reaches beyond GANs — color transfer between images compares pixel distributions by earth-mover cost, and document retrieval systems embed histograms of words and rank them by transport distance.

11.9 WGAN: Wasserstein Generative Adversarial Network

11.9.1 Kantorovich-Rubinstein Duality and Lipschitz Functions

Direct evaluation of W is intractable, but a certain duality rescues it: the Kantorovich-Rubinstein duality. Its derivation belongs to real analysis, not to this course — consult a real analysis text for the proof — but the statement is compact. The infimum over transport plans can be rewritten as a supremum over functions:

Plain words: sample $x$ from the real distribution and evaluate some function $f$; sample from the generated distribution and evaluate the same $f$. Take the difference of the two averages. Search over the allowed class of functions for the one making this difference largest. That largest difference equals the Wasserstein distance.

Why should such a duality hold? A one-line intuition: a gentle function cannot spike up only where reals live and dive only where fakes live unless mass genuinely had to be carried between them — the smoothest possible "price tag" landscape over space encodes exactly how much hauling separates the two piles. The proof makes this rigorous; using the statement needs no real analysis.

The allowed class is the 1-Lipschitz functions. A function $f$ is K-Lipschitz continuous if there exists a real constant $K\ge 0$ such that for every pair of points:

Setting $K=1$ gives the 1-Lipschitz family: functions whose steepness nowhere exceeds one. Two quick membership checks: $f(x)=2x$ fails (slope 2 everywhere), while $f(x)=\sin x$ passes, since its slope $|\cos x|$ never tops 1; and $f(x)=x^2$ fails outright because its slope grows without bound. Crucially this is not one function but a vast class — the optimization searches within that class for the right member, and the search happens by adjusting neural network weights.

11.9.2 The WGAN Objective and the Critic

Install the dual form directly into the GAN template. The generator minimizes, the function maximizes, and the function must stay 1-Lipschitz:

Compare with the original formulation's tangle of $\log D(x)$ and $\log (1-D(G(z)))$ terms — this objective is much simpler. Terminology shifts too: $D$ is no longer called the discriminator but the critic. The rename is earned. A discriminator answers a two-value question — real or fake — squeezing everything into probabilities between 0 and 1. A critic scores how different two distributions are, on an unbounded scale: high output when things differ, low when they match. The critic must be learned — among all 1-Lipschitz functions, pick whichever maximizes the gap, shaping it by updating weights.

11.9.3 Enforcing the Constraint: Weight Clipping and Training Pseudocode

How do you force a neural network to respect the Lipschitz bound? The original answer is blunt: weight clipping. Constrain all critic weights to live in a compact interval $[-c,c]$ around zero, with $c$ a small constant. After every update, squash any weight outside the box back onto its edge. Clipped weights cannot have arbitrarily steep input-output slopes, which keeps the critic inside the Lipschitz class.

The full training loop, with symbols named. Let $\alpha$ be the learning rate. Let $c$ be the clipping constant. Let $m=64$ be the batch size and $n_{\text{critic}}=5$ the number of critic iterations per generator iteration. Let $w_0$ denote the initial critic parameters and ${\theta }_0$ the initial generator parameters.

Much of this mirrors the original GAN pseudocode — alternate players, train the judge more often than the builder — but the objective is different, and so is the optimizer choice. Note the asymmetry by design: several critic steps per single generator step, because the critic must stay near its own optimum for its score to mean "distance."

Complexity and cost: each generator step costs five critic steps plus clipping, so WGAN training runs roughly five times the critic compute of a vanilla GAN step-for-step; the payoff is that far fewer total steps are needed before samples look reasonable.

When to use / alternatives: use the WGAN objective when vanilla GAN training oscillates or collapses; alternatives are the improved-techniques toolbox of Section 11.6 or the gradient penalty below, which replaces clipping entirely.

11.9.4 Why WGAN Trains Better

Three empirical behaviors justify the machinery.

First, gradients survive bad starts. Watch the critic's values as training begins, when the generated distribution sits far from the real one. The critic's output still separates cleanly and its curve changes smoothly with distance, so the generator receives a strong, meaningful gradient signal even in the worst regime. Contrast the standard GAN: a confident discriminator saturates, outputting near-zero on fakes everywhere, so differentiating its loss yields vanishing gradients for the fake samples precisely when the generator most needs direction. The critic replaces collapse with proportionality.

Second, the distance tracks quality. With the Wasserstein estimate in use, as the perceptual quality of generated data climbs, the measured Wasserstein distance falls. That correlation fails under JSD, whose plot stays flat and only occasionally dips without any link to visible quality. Distance here is correlated with sample quality — a diagnostic you can actually read. Side-by-side sample grids made the case visibly: early WGAN output already showed clarity that the baseline never reached.

Third, stability under architecture edits. Tiny architectural tweaks to a DCGAN flip how results come out — drop batch normalization and the outcome jumps completely. WGAN degrades gracefully under the same perturbations; it is insensitive to architectural choices in a way DCGAN never was.

11.9.5 Open Problems: Clipping Side Effects and What Comes Next

Nothing is free here either. Weight clipping enforces Lipschitz-ness crudely: once training settles, weights pile up at the rails $-c$ and $+c$, almost none resting between them. The capacity of the weight range goes unleveraged — the network cannot express fine distinctions when every weight is pinned at an extreme.

The announced remedy is the gradient penalty. Instead of clipping, take the gradient of the critic function itself and penalize its magnitude directly, nudging the critic toward slope one without crushing weights onto rails. That improvement opens the next meeting, which continues through WGAN variants and the wider GAN architecture tour — conditional GAN, CycleGAN, and StyleGAN among the popular designs. After that, the course advances to diffusion models and energy-based models for data generation.

Real-world placement: WGAN's readable loss became a monitoring standard — teams plot the critic score during training the way others plot validation accuracy, because unlike most generative losses it moves when quality moves.