Variants of Generative Adversarial Networks

These notes cover the main branches of the GAN family built on the Wasserstein idea. First comes the refresher: how the Wasserstein distance and the critic replace the shaky original GAN loss. Then three ways to enforce the smoothness demand that makes the Wasserstein setup work: weight clipping, the gradient penalty, and spectral normalization. Two quality boosters follow — self-attention, which lets every image position consult every other position, and progressive growing, which builds huge images resolution by resolution.

The second half turns random generation into directed generation: the conditional GAN steers samples with class labels, Pix2Pix converts one image into another rendering of it, the auxiliary classifier GAN adds label prediction to the judge, and CycleGAN learns style changes without paired examples. Each section keeps the spoken explanation next to every formula, so you can trace how each equation was introduced and why each design exists.

12.1 WGAN Refresher: Wasserstein Distance and the Critic

Before new material, we rebuild the base. The Wasserstein GAN replaces the unstable divergence used in the original GAN with a smoother way to measure how far two probability distributions sit from each other. Everything later in these notes — clipping, gradient penalty, spectral normalization — depends on this one shift.

12.1.1 Wasserstein Distance and Duality

The Wasserstein distance is a distance metric on probability densities. It is well behaved and stable, and it can measure when two probability densities differ. The picture to hold in mind is earth moving: the distance is about "finding the best possible plan of converting one particular discrete probability distribution to another discrete probability distribution". You decide how much mass to move from each pile of the first distribution to each pile of the second, and you pick the cheapest such plan.

Now formalize this. Let $P_r(x=i)$ be the real distribution over bins and $q(x=j)$ the target distribution, both defined over $K$ discrete bins. Let $C_{ij}$ be the cost of moving one unit of mass from bin $i$ to bin $j$ — a natural choice is the index gap $|i-j|$. A transport plan is a matrix $P$, where entry $P_{ij}$ records how much mass travels from bin $i$ to bin $j$. The Wasserstein distance is the price of the best plan:

$$W(P_r,q)=\underset{P}{\min }\sum _{i=1}^K\sum _{j=1}^K{P}_{ij}\cdot C_{ij}$$

subject to three constraints that make $P$ a legal shipping schedule:

$$\sum _j{P}_{ij}=P_r(x=i),\sum _i{P}_{ij}=q(x=j),P_{ij}\ge 0$$

Read the constraints out loud: each row must ship out exactly what bin $i$ owns; each column must deliver exactly what bin $j$ expects; no route can carry negative cargo. Every symbol here is a plain quantity — nothing exotic. The minimization picks the single cheapest schedule among all schedules that satisfy those rules.

This plan search is costly. With $n$ support points there are $n^2$ plan entries to decide, so the computation grows on the order of $n^2$ — "order n square for sure", as it was put. Solving a full transport problem inside a training loop, which runs thousands of updates, is impractical.

The escape is a duality principle from linear programming. It says the same value can be reached from another direction: instead of planning transports, look for one scoring function whose gap between the two distributions equals the distance. In its primal form the transport problem minimizes a cost over plans $p$; the dual problem maximizes a score over per-bin values $f_i$, subject to the rule that adjacent scores cannot differ by more than one. For the continuous version used in GANs, the dual reads:

$$W(P_r,P_g)=\underset{f}{\max }{\mathbb{E}}_{x\sim P_r}[f(x)]-{\mathbb{E}}_{x\sim P_g}[f(x)]$$

where the maximization runs over functions whose slope never exceeds one in absolute value. That slope cap is exactly the "adjacent values cannot change by more than one" rule lifted from bins to a continuum. The scoring function is what the GAN world calls the discriminator, and in the Wasserstein setting it is called the critic. Training a network to be the critic trains it to compute the distance — without ever building a transport plan.

12.1.2 The Critic Function and Its Value Range

The critic gives a high value or a low value — a continuous value — depending on how good the generated data is. Unlike the classic discriminator output, this number does not live between zero and one. It is "a number between minus infinity to plus infinity". Nothing forces it to be a straight-line function either; the point is only that it spans negative values up to positive values.

That slope restriction has a name. A K-Lipschitz function satisfies

$$|D(x_1)-D(x_2)|\le K|x_1-x_2|$$

for all input pairs $x_1,x_2$. Here $D$ is the critic function, $x_1$ and $x_2$ are any two inputs, and $K$ bounds how fast the output may change relative to input change. Spoken form: "the derivative of the function, if it is less than or equal to one, then it is one Lipschitz; it is K Lipschitz if it is less than or equal to K in absolute value". The one-Lipschitz case is simply $K=1$: tilt your hand holding a pencil along the critic's graph, and the steepest tilt you can meet is 45 degrees.

Visual intuition: plot two narrow, non-overlapping bell curves on the same axes — horizontal axis is the data value, vertical axis is density. The original GAN's loss goes flat between them: zero gradient, no learning signal. Now draw a gently rising line above them with slope at most 1, higher over the right curve than the left one. That line is the critic. Because its height difference encodes the distance, sliding either curve changes the line's tilt, and gradients keep flowing even across the empty gap.

Why does this matter for training? A Lipschitz critic has well-behaved derivative values almost everywhere, and those derivatives are what flow into the generator update. Bounded slopes mean the generator always receives a usable push toward realism — this is why the Wasserstein setup trains smoothly where the old setup stalled.

12.1.3 The WGAN Objective

Here is the value function optimized in WGAN training, with $\tilde{x}=G(z)$ a generated sample, $x$ a real training sample, $z$ a latent vector drawn from a noise distribution $p_z$, and $G$ the generator network:

$$\underset{G}{\min }\underset{D\in \text{1-Lip}}{\max }{\mathbb{E}}_{x\sim P_r}[D(x)]-{\mathbb{E}}_{\tilde{x}\sim P_g}[D(\tilde{x})]$$

Every symbol: $P_r$ is the real data distribution, $P_g$ is the distribution induced by the generator, and $D\in \text{1-Lip}$ means the search over critics is restricted to one-Lipschitz functions. The spoken walk-through, rephrased: minimize the whole expression with respect to $G$; for real data, $D(x)$ should return the high value, and for fake data that number will be small, so you subtract a small number from a large number — the goal is to maximize with respect to $D$.

Each iteration plays one move per player. The critic climbs toward the true distance; the generator then walks down a landscape that always points it toward the real data. Neither player ever sees a dead zone with no gradient.

Why bother switching divergences? Using Jensen-Shannon divergence or KL divergence to compare the two distributions is what triggers instability and mode collapse, where the generator produces only a few kinds of outputs — imagine a digit generator that draws 1s and 7s forever and never attempts an 8. Those divergences behave badly when the two distributions barely overlap: the measured difference shoots to a constant and the gradient dies. The Wasserstein measure alleviates that problem to a great extent, gives much better control over mode collapse, and stays forgiving about architectural choices.

Real-world: this stability is why WGAN-family models became the default backbone for many image-generation pipelines, from super-resolution systems to synthetic-data generators used in medical imaging research. Any product that needs reliable day-after-day GAN training inherited this design decision.

12.2 Weight Clipping in Standard WGAN

The first WGAN enforced the one-Lipschitz demand with a blunt tool: squeeze the weights. It worked well enough to prove the idea, and it exposed exactly why a smarter tool was needed. Even the authors of the original paper said at the time that clipping was not the right long-term answer — they simply lacked something better.

12.2.1 Clipping Weights to Enforce One-Lipschitz

After every critic update, clamp every weight into a fixed box:

$$w\leftarrow \mathrm{clip}(w,-c,c)$$

Here $w$ is any weight of the critic and $c$ is the weight clipping parameter — a small positive constant you pick. The clip operator returns $w$ unchanged when $-c\le w\le c$, snaps up to $+c$ when $w>c$, and snaps down to $-c$ when $w<-c$. When you implement standard WGAN, this parameter $c$ is the knob you set.

Why does boxing weights control slope? A network's output changes when its inputs change through a chain of weighted multiplications. If each weight is small, each multiplication can only stretch its input by a bounded amount, so the composed function cannot swing steeply — with all weights boxed, the critic function cannot grow an arbitrarily steep tilt, and the $K$-Lipschitz condition holds in practice.

12.2.2 Gradient Control and Depth Limits

Why must $c$ be small? One diagnostic plot makes the danger visible. Its horizontal axis lists the discriminator layers from input to output; its vertical axis shows the gradient magnitude on a log scale; one curve is drawn per value of $c$. With large boxes, the curves climb out of control as they cross layers — the exploding-gradient regime, where each successive layer multiplies the problem. So you must use a small value of $c$, and you must also restrict the number of layers in the critic. Deep critics plus loose boxes invite the same explosion, because more layers mean more multiplicative stretch opportunities stacked in a row.

The original WGAN paper used a box of roughly $\pm 0.01$ — tight enough that gradients stay tame, loose enough that some learning still happens.

12.2.3 The Capacity Problem at the Two Extremes

Clipping has a hidden flaw that people observed during training. You clip so that $-c$ and $+c$ act as minimum and maximum values for the weights once training ends. But the weights do not spread out evenly inside the box: they pile up pressed against the two walls. Almost no weights sit at intermediate values.

Visual intuition: draw a histogram of the trained weights with weight value on the horizontal axis and count on the vertical axis. A healthy network gives a hill-shaped histogram centered near zero. A clipped WGAN gives two tall spikes at exactly $-c$ and $+c$ with an empty valley between them — the training pushes weights against the walls, and the clip keeps snapping them back there.

The side effect shows up in the metric that matters: your FID score will not be low for vanilla clipped WGAN. The FID score (Fréchet Inception Distance) is a realism measure where lower is better. It passes many real images and many generated images through a pretrained image network, summarizes each set by a mean vector and a covariance matrix of internal features, and scores how far apart those two statistical fingerprints sit. A generator producing rich, varied, realistic samples drives the score down; a capacity-starved critic fails to guide the generator there, so the score stays high.

Real-world: tuning such knobs is treated as job training. In the workplace you often need not just to choose the right technique but to produce results with it — knowing how to turn the knobs transfers directly to day-to-day work, where the same skill decides whether a model ships or stalls.

12.3 WGAN with Gradient Penalty (WGAN-GP)

The gradient penalty replaces weight boxing with a soft, mathematically grounded demand on slopes. Instead of squeezing weights and hoping the slope behaves, you measure the slope directly on meaningful points and push it toward exactly one. It is a significant piece of work, and it is the version you will implement in the assignment.

12.3.1 The Convex Combination Input

Place the two distributions side by side: one cloud is the real training images, the other is the generated images. The penalty lives on the straight line joining a real point to a generated point. Concretely, draw one real sample $x$ from the training set, one generated sample $\tilde{x}=G(z)$, and mix them:

$$\hat{x}=ϵx+(1-ϵ)\tilde{x},ϵ\sim U[0,1]$$

Here $\hat{x}$ is the mixed point, and $ϵ$ is a fresh uniform draw between zero and one for each mixed sample — $U[0,1]$ means every value in that interval is equally likely. When $ϵ=1$ you stand on the real image; when $ϵ=0$ you stand on the generated one; in between you stand somewhere on the segment connecting them. Because the two weights $ϵ$ and $1-ϵ$ are non-negative and add to one, this mixture is called a convex combination — a weighted average that can never leave the segment.

A stressed point about geometry: the figure showing the two clouds and the connecting segment is drawn in two dimensions with coordinates $x_1,x_2$, but real images live in a very high dimensional space — even 32 by 32 images give a 1024-dimensional input. The "straight line" between two samples is a line through that high-dimensional space, not through picture space.

12.3.2 The Gradient Penalty Term and the Full Objective

The penalty asks one thing of the critic: on those mixed points, the gradient magnitude should sit near one. Written out, with $\lambda$ a weighting coefficient:

$${\mathbb{E}}_{\hat{x}}\left[{\left(\Vert {\nabla }_{\hat{x}}D(\hat{x}){\Vert }_2-1\right)}^2\right]$$

Every symbol: ${\nabla }_{\hat{x}}D(\hat{x})$ is the gradient of the critic output with respect to its input $\hat{x}$ — how fast the score changes as you nudge the mixed image; $\Vert \cdot {\Vert }_2$ is the L2 norm, which turns that gradient vector into a single magnitude; squaring makes the penalty zero exactly when the magnitude equals one and grows smoothly on both sides of that target. Spoken form, rephrased: take the partial derivative of $D$ with respect to $\hat{x}$; take its absolute value — or L2 norm — the gradient magnitude with respect to the input, where the input is a convex combination of the training data and a generated sample; the penalty value goes down if the average of the gradient magnitude becomes close to one.

Put together, the critic minimizes the original critic loss plus the penalty:

$$L_D=\underset{\text{original critic loss}}{\underbrace{{\mathbb{E}}_{\tilde{x}}[D(\tilde{x})]-{\mathbb{E}}_x[D(x)]}}+\lambda {\mathbb{E}}_{\hat{x}}\left[{\left(\Vert {\nabla }_{\hat{x}}D(\hat{x}){\Vert }_2-1\right)}^2\right]$$

The first bracket is small when the critic separates real from generated well — high $D(x)$, low $D(\tilde{x})$. The second bracket vanishes when the average gradient magnitude over all mixed points is about one; otherwise it grows quadratically and pulls $D$ back whenever the slope drifts off one. Note the sign flip relative to Section 12.1: here the expression is written as a loss to minimize, so the terms are reversed. The generator then maximizes the critic's score on fakes:

$$L_G=-{\mathbb{E}}_{\tilde{x}}[D(\tilde{x})]$$

A quick numeric feel for the penalty: if the measured gradient norm on a mixed point is $1.2$, that point contributes $(1.2-1{)}^2=0.04$. If the norm collapses to $0.5$, the contribution jumps to $(0.5-1{)}^2=0.25$ — six times larger, because flat critics are just as bad as steep ones.

12.3.3 Pseudo-code Walkthrough of the Training Loop

You will implement this loop in the assignment, so here is the full sequence with every quantity named.

12.3.4 Practical Rules: Batch Normalization and Architecture Freedom

One rule surprises everyone: there is no batch normalization in the critic. Batch normalization rescales each layer's values using statistics computed over the whole mini-batch, but the penalty judges each instance by its own slope within the group — mixing batch-level averages into those judgments corrupts what the penalty measures. People observed empirically that adding it does not work well, so the critic simply omits it. The generator remains free to use batch normalization as usual.

Second rule: WGAN-GP is forgiving about architecture. You can vary activation functions, vary the depth of $G$ and $D$, and vary the base filter counts — the convolution widths used in generator and discriminator — and training still works. Compare that with the plain DCGAN recipe, where a long prescription applies: strided convolutions, batch normalization in specific spots, none at the end of the generator or the start of the discriminator. Deviate from that recipe and results degrade. The gradient penalty eliminates that brittleness broadly. Vanilla WGAN sits in between: it is fragile, and shuffling components can make it fail outright, because clipping starves the critic of usable capacity.

Dataset context matters when judging results. The showcase samples for this method come from LSUN, a dataset of indoor scene pictures — complex because objects occupy a spatial organization inside a room. CIFAR-10, the assignment dataset, is relatively simple: one object against a background. Expect WGAN-GP to produce solid samples at CIFAR-10 difficulty.

The price is time. The penalty computation itself is long: build the convex combinations, push them through the critic, observe outputs, then differentiate through all of it. Gradient-penalized training takes visibly longer per step than clipped training — accept slower steps as the price of principled constraint enforcement.

12.3.5 Reproducibility Concerns

These methods start from random initial points, so run-to-run variation is real, and you should plan for it. The everyday illustration: ask ChatGPT the same prompt with memory turned off, on different days and at different times, and the answers will not be exact replicas of each other. That does not make one answer right and the other wrong; randomness is baked into how such systems learn. For industrial deployments, though, customers expect dependable behavior, so teams add heuristics and engineering controls — fixed random seeds, pinned configurations — to show reproducibility on demand.

12.3.6 Student Questions and Answers

12.4 Spectral Normalization GAN (SNGAN)

Clipping was the first mechanism for the Lipschitz demand, and even its inventors said at the time that it was not the right way to achieve it — they lacked a better one. The gradient penalty is principled but slow. Spectral normalization is the third route: constrain the weight matrices themselves, layer by layer, so the critic physically cannot have a steep slope. Instead of punishing violations after they happen, this design prevents them by construction.

12.4.1 From the GAN Objective to the Lipschitz Constraint

Start from the original GAN formulation for reference:

$$\underset{G}{\min }\underset{D}{\max }{\mathbb{E}}_{x\sim P_{\text{data}}}[\log D(x)]+{\mathbb{E}}_{z\sim p_z}[\log (1-D(G(z)))]$$

Spoken: "for the actual training data, log of D(x)... and for the data coming out of the generator, log of one minus D... you want to maximize with respect to D and minimize with respect to G." Here $P_{\text{data}}$ is the real data distribution, $p_z$ the latent noise distribution, and $G(z)$ the generated sample fed to the discriminator $D$.

The WGAN reformulation restricts the scorer to a bounded-slope class:

$$\underset{w:\mathrm{Lip}(f_w)\le K}{\max }{\mathbb{E}}_x[f_w(x)]-{\mathbb{E}}_z[f_w(G(z))]$$

where $f_w$ is the critic network with weights $w$, and the Lipschitz constant $\mathrm{Lip}(f_w)$ — the steepest possible local slope — must stay at most $K$, a fixed bound. Spoken: "F is the class of functions that implements this critic; this Lipschitz value for that function has got to be less than or equal to K, for a fixed K."

What pins down $K$ is the slope definition. For two nearby inputs,

$$\left|\frac{f_w(x)-f_w(x^{\prime })}{x-x^{\prime }}\right|\le K$$

and in the infinitesimal limit the finite difference on the left becomes the derivative: "in the infinitesimal sense, this finite difference, numerator over denominator, can be replaced by the partial derivative." So the whole game is bounding the critic's derivative. Spectral normalization does it matrix by matrix.

12.4.2 Spectral Norm Definition

Recall from the linear-algebra part of the mathematical foundations (MFML) course: the spectral norm of a matrix $A$ tracks its largest eigenvalue. The precise statement, guarding against a zero denominator:

$$\sigma (A)=\underset{h\ne 0}{\max }\frac{\Vert Ah{\Vert }_2}{\Vert h{\Vert }_2}=\underset{\Vert h{\Vert }_2\le 1}{\max }\Vert Ah{\Vert }_2$$

Here $A$ is a weight matrix, $h$ is an input vector to that layer, and $\Vert \cdot {\Vert }_2$ is the Euclidean norm (the usual square root of the sum of squared entries). Spoken: "spectral norm of A is given by the dot product of A times h; you take the L2 norm of it and divide by the L2 norm of h, with h not equal to zero... for magnitude of h less than or equal to one." During the lecture the first wording was "sum of eigenvalues", corrected midstream to "not sum — the eigenvalue associated with the largest eigenvalue".

Why does this control slope? Take a linear layer $g(h)=Wh$ with input vector $h$ and weight matrix $W$. Then

$$\mathrm{Lip}(g)=\sigma (W)$$

The layer's Lipschitz constant equals the spectral norm of its weight matrix — the loudest stretch the layer can apply to any direction. If the activation function's derivative never exceeds one in absolute value, the activation adds no further stretch, and the Lipschitz condition survives it.

12.4.3 Composition Across Layers

Deep networks chain layers, and slopes multiply along the chain. If layer $l$ applies weights $W_l$ followed by activation $a_l$, then the whole network obeys

$$\mathrm{Lip}(a_L\circ g_L\circ \cdots \circ a_1\circ g_1)\le \prod _{l=1}^L\mathrm{Lip}(a_l)\sigma (W_l)$$

Spoken: "the Lipschitz value associated with the overall combination of the functions implemented by the layers is going to be less than or equal to the multiplication of the Lipschitz values of the individual pieces." The composition symbol $a_L\circ g_L\circ \cdots \circ a_1\circ g_1$ just means: run the stages in order, layer 1 first. Starting from input $h_0$, each stage maps $h_{l-1}$ through $W_l{h}_{l-1}$ into the total inputs of layer $l$'s nodes, then through activation $a_l$ to produce $h_l$.

So the recipe is mechanical: for every weight matrix in the discriminator, compute its spectral norm, and divide all its weights by that number. After division, every $\sigma (W_l)$ equals one — a matrix divided by its own loudest stretch can stretch nothing louder than one — and the composed network lands at Lipschitz constant one. Normalizing each $W_l$ tames every stage in that chain.

12.4.4 Power Method Computation

Computing a spectral norm sounds heavy — a full eigendecomposition per layer per step would be crushing. How is it done cheaply? By the power method — an iterative scheme that estimates the largest-magnitude stretch direction of a matrix without ever decomposing it.

You initialize a random vector ${\tilde{u}}_l$ of the appropriate dimension, sampled from an isotropic Gaussian (every direction equally likely), then repeatedly push it through the matrix and renormalize:

$${\tilde{v}}_l=\frac{W_l^{\top }{\tilde{u}}_l}{\Vert W_l^{\top }{\tilde{u}}_l{\Vert }_2},{\tilde{u}}_l\leftarrow \frac{W_l{\tilde{v}}_l}{\Vert W_l{\tilde{v}}_l{\Vert }_2}$$

Each pass multiplies by the matrix, which amplifies the dominant direction relative to all others; renormalizing keeps lengths at one so nothing overflows. Repeat a few rounds, and the pair $({\tilde{u}}_l,{\tilde{v}}_l)$ converges toward the dominant directions. The scalar estimate falls out as

$${\sigma }_l(W_l)={\tilde{u}}_l^{\top }{W}_l{\tilde{v}}_l$$

which is a scalar: one row-vector, one matrix, one column-vector.

Only two or three power iterations are needed to reach a reasonable degree of precision, which is why the whole scheme earns the label "fast computation". In practice SNGAN even reuses the previous step's direction vector as the starting point, because it barely moved between updates.

12.4.5 Weight Update After Normalization

During training, each critic step uses the spectrally normalized weights in the forward pass and then takes an ordinary gradient step on them:

$$W_l\leftarrow {\bar{W}}_l-\eta {\nabla }_{{\bar{W}}_l}L$$

Here ${\bar{W}}_l=W_l/{\sigma }_l(W_l)$ is the normalized weight matrix, $L$ is the critic loss, and $\eta$ is the learning-rate step size. Spoken: "the new weights become the old weights minus epsilon times the gradient of the loss function with respect to the normalized weights."

12.4.6 Hinge Loss in SNGAN

SNGAN changes the loss, not just the constraint. It adopts the hinge loss, the same loss family used in SVMs (support vector machines), and it behaves differently from the log losses of WGAN and WGAN-GP:

$$L_D=-{\mathbb{E}}_x[\min (0,D(x)-1)]-{\mathbb{E}}_{\tilde{x}}[\min (0,-D(\tilde{x})-1)],L_G=-{\mathbb{E}}_{\tilde{x}}[D(\tilde{x})]$$

Here $x$ ranges over real samples, $\tilde{x}=G(z)$ over generated ones, and the target scores are $+1$ for real and $-1$ for fake. Walk through the values one state at a time.

12.4.7 Comparing the Four Designs

Line the designs up in order.

A plain GAN pays for its simple loss with instability and mode collapse. WGAN switches to the Wasserstein distance, gains stable training, curbs mode collapse, and improves FID scores — but its weight clipping piles weights at the box edges and wastes critic capacity, so results stay mediocre. WGAN-GP fixes the constraint properly: compute gradient magnitudes on convex combinations of training and generated data, and penalize any deviation of that magnitude from one. This produces significantly better results at the cost of computational complexity. SNGAN prevents the violation instead of penalizing it: divide each weight matrix in the critic by its spectral norm, estimated by the fast power method, gaining a computational benefit over WGAN-GP — and it swaps the loss to the hinge form. When to pick which: if you can afford compute, WGAN-GP; if per-step speed matters, SNGAN.

12.4.8 Student Questions and Answers

12.5 Self-Attention GAN (SAGAN)

Convolution windows see only local neighborhoods. SAGAN bolts an attention mechanism onto the GAN stack so every output position can consult every other position, weighted by relevance.

12.5.1 Attention Over Feature Maps

Attention first appeared in deep networks for sequences: the prediction gets influenced by the past context, and the context that matters more receives more weight. SAGAN imports the same idea into images. The feature maps flowing out of each convolutional layer — in both the generator and the discriminator — become the raw material for attention maps. Visualizing those maps shows bright spots where the model attends strongly and dim regions it mostly ignores.

Because the model can pull information from relevant distant locations instead of just the immediate window, results often come out much better: proper weight goes to the information sitting in the wider neighborhood. Long-range coherence — matched sky colors, consistent object parts, aligned textures — improves visibly.

12.5.2 Query, Key, Value Computation

The engine is the query-key-value trio. From the layer's features, learned projections produce a query $Q$, a key $K$, and a value $V$:

$$o=\mathrm{softmax}(Q{K}^{\top })V$$

Spoken: "you use the QKT concept — Q is the query, K is the key, V is the value — and you implement that with softmax. Out of softmax we get different weights that you multiply element by element with the feature vector to create the self-attention feature maps."

Shapes make this precise. Suppose a feature map has $N$ spatial positions (for a height-by-width grid, $N$ counts all cells) and each position carries a feature vector. Learned matrices project each position into three roles: a query vector ("what am I looking for"), a key vector ("what do I offer"), and a value vector ("what I pass along if chosen"). The product $Q{K}^{\top }$ builds an $N\times N$ table whose entry $(i,j)$ scores how much position $i$ should listen to position $j$. The softmax turns each row of that table into positive weights adding to one — a relevance budget per position. Multiplying by $V$ gathers the attended values into the output $o$: each output position becomes a relevance-weighted blend of value vectors from everywhere in the image.

12.5.3 Residual Output with Gamma

The attention output merges back through a residual gate:

$$y=x+\gamma o$$

Here $x$ is the layer's input, $o$ is the attention output, and $\gamma$ is a learned scale — a single number trained by gradient descent, initialized at zero. Spoken: "your output becomes nothing but the input to the layer plus gamma times the O." Starting at $\gamma =0$, training begins with attention nearly switched off — the network behaves like a plain convolutional stack — and $\gamma$ grows only as the task rewards long-range mixing. This keeps early training stable while leaving the door open for global context. The result: the output of a convolution layer now also carries information about the broader context, not just the local patch.

12.5.4 Where SAGAN Fits

Everything else stays on the same rails as before: spectral normalization applied to both generator and discriminator weights, hinge-form losses, and self-attention modules inside both networks. The headline claim: SAGAN was the first variant of the GAN to produce good unconditional full-image samples at ImageNet scale. ImageNet is the standard large-scale image corpus — many million labeled pictures, with the widely used classification benchmark built on 1000 everyday classes (the lecture's slides quoted a smaller class figure; the canonical benchmark count is 1000). "Unconditional" means generation is not steered toward any particular class: the model draws random images that look like members of the image population. Against CIFAR-10 — a far smaller dataset with very few classes — the jump matters less, but for creating data similar to ImageNet yet distinct from it, SAGAN beat every earlier variant discussed. Reported FID scores differ noticeably across SAGAN versions, which is why benchmark tables list several.

Real-world: long-range consistency is exactly what product pipelines need when synthesizing scenes — furniture catalogs generate whole rooms where floor lighting must agree across the frame, and game studios build texture generators whose distant patches must not clash. Self-attention modules from this line later became standard equipment in the vision backbones behind such systems.

12.6 Progressive Growing GAN

How do you synthesize a 1024 by 1024 image when your generator starts from a tiny random vector? Do not train the whole monster at once. Grow it.

12.6.1 Building a Resolution Pyramid

Suppose the target is large images, say 512 or 1024 by 1024. From the training set of full-resolution images, manufacture smaller copies: 512 by 512, 256 by 256, and so on, down to an extreme case of 4 by 4. Simple subsampling does it — mean pooling with stride equal to 2 progressively shrinks the images, halving each side per stage. You end up with a pyramid of the same content at every resolution: nine levels connect 1024 down to 4 (1024 → 512 → 256 → 128 → 64 → 32 → 16 → 8 → 4), each step cutting both width and height in half.

12.6.2 Growing the Networks Step by Step

First train a small game — a compact generator-discriminator pair — that works well on the 4 by 4 versions of the images. Then extend: add layers to the generator and discriminator so the next pair handles 8 by 8, train it, continue through 16 by 16, and onward until the full 1024 by 1024 pair stands. The generator pathway starts from a small random latent vector — about 128 values spoken of as the input size; the key point is that this input is a flat, low-dimensional vector, not an array of pixels — and expands through feature maps of 4 by 4 with 1024 filters, then 8 by 8 with 512 filters, doubling resolution while halving filter counts as it climbs. So the earliest layer already works on tiny spatial grids but with many channels, and each growth stage trades channel count for spatial detail.

One refinement made this ladder stable in practice: new layers fade in gradually. When a higher-resolution stage is added, its output initially blends with a simple upscaled copy of the previous stage's result through a residual connection, and the new layers take over only as their influence ramps up. The network never experiences a harsh architectural jump.

Visual intuition: imagine two staircases rising together, one for the generator (small blurry grid growing into a sharp large image) and one for the discriminator (large sharp input shrinking back to a single real-or-fake verdict). Each staircase step on the left must match a step on the right, or the game falls out of balance.

The payoff is simple: this process works much better and faster than trying to train a one-shot generator that jumps straight from a small latent vector to a 1024 by 1024 image. Training progresses 4 by 4, then 8 by 8, then 16 by 16, finally 1024 by 1024 — a genuinely powerful technique that delivers good results quickly. Early resolutions teach global layout cheaply; late resolutions spend capacity only on fine detail.

Real-world: progressive growing is what first made photorealistic human-face generation practical — the face images produced by this family are famous enough that synthetic faces built this way have been used as stock avatars and privacy-preserving profile photos.

12.7 Conditional GAN (CGAN)

Random generation is fun; directed generation is useful. The conditional GAN adds a steering wheel: generate images belonging to one particular class of the training data.

12.7.1 Conditioning on Class Labels

Take CIFAR-10 as the running example: 50,000 training images spread across 10 classes — airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck. That class information is gold at generation time — feed it to the generator so it creates samples from the requested class.

Mechanically, the class label travels with the noise: alongside the latent vector $z$, the model receives the label, encoded as a vector. Labels like cat or dog are symbolic words, and networks compute on numbers, so they need an embedding — a numeric vector standing in for the category. The simplest embedding is one-hot encoding: with 10 classes, each label becomes a 10-slot vector holding zeros everywhere except a single 1 at its own slot. Concatenate that one-hot label onto $z$, or otherwise fuse the two, and hand the pair to the generator.

The discriminator gets the label treatment too: it sees the label along with whatever image it judges, real or generated. Its job becomes sharper — "is this a real horse?" rather than "is this real?" In implementation terms, the label can be appended to the input for flat data, or reshaped into a map of values and added as an extra channel alongside the image channels for convolutional judges. So the whole pipeline restricts itself class by class — the class acts as the condition inside the generator itself. Training then updates both models: the discriminator learns real-versus-fake classification per label, and the updates improve the convolutional filter weights in both the generator and the discriminator.

12.7.2 Conditional Loss Functions

With $m$ samples in a mini-batch, $x_i$ a real image, $y_i$ its class label, $z_i$ a latent vector, and $c_i$ the conditioning label fed to the generator, the real-sample loss for the discriminator is

$$L_D^{\text{real}}=-\frac{1}{m}\sum _{i=1}^m\log D(x_i,y_i)$$

Spoken, rephrased: average minus the log of the judge's score on true images paired with their true labels — the minus sign flips the small negative logarithm into a positive loss, so minimizing pushes scores toward one. The fake-sample loss passes the label through the generator and the judge:

$$L_D^{\text{fake}}=-\frac{1}{m}\sum _{i=1}^m\log (1-D(G(z_i\mid c_i),y_i))$$

The notation $G(z_i\mid c_i)$ reads "the sample generated from noise $z_i$ given condition $c_i$" — the vertical bar marks which information steers generation.

Trace the arithmetic logic once more in words: on a convincing fake, $D$ outputs a value near zero — not near one — so $1-D$ sits close to one, its logarithm sits close to zero, and the leading minus leaves only a small number to add. Minimizing $L_G$ teaches $G$ to produce images that both look real and carry the demanded label; the label inside the loss is what makes the demand stick.

Real-world: class-conditioned generation powers data augmentation for rare categories — self-driving simulators request exactly "truck in fog" style samples to balance training sets, and designers use conditioned generators to draft variations of one product category without touching others.

12.8 Pix2Pix GAN

Every GAN so far started from noise. Pix2Pix starts from an image and produces a different rendering of it — the flagship of the conditional family.

12.8.1 Image-to-Image Translation

The input to Pix2Pix is not a random vector. It is a version of an image, and the goal is to change the type of that image — this task shape is called image-to-image translation. Canonical cases: a grayscale photograph becomes a realistic color photograph; the outline of a sneaker shoe becomes the actual rendered sneaker; a silhouette fills in the object and background. Outline images have a more technical name: edge images. The mapping idea extends beyond pictures — feeding an English sentence and receiving the German counterpart fits the same shape, source in, closely related target out. That sentence view is exactly how neural machine translation works, which is why the two tasks share a name.

The generator $G$ takes an image and creates another image out of it; the discriminator tries to expose the synthesized one as fake, exactly as in a standard GAN. It counts as a conditional GAN because the input image plays the role the class label played before — the condition the generator is conditioned on.

12.8.2 Pair-Based Discrimination

Training leans on pairs. The discriminator receives two images together: the generated image and its corresponding training counterpart. Its verdict applies to the pair as a whole — a real pair earns a high value, a fabricated pair earns a low value. Picture the training bank: real images of cold-drink bottles, cats, dogs, and so on, each stored alongside its alternate version, black-and-white or edge-rendered, aligned pixel by pixel. The generator learns to render a realistic version of whatever input it receives, where "realistic" means matching the reality captured by the training pairs.

This strictness is what makes paired data both powerful and expensive to collect: every training outline must come with its own photographed twin.

12.8.3 Adversarial Loss Plus L1 Reconstruction

The adversarial piece scores pairs, with $x$ the input image, $y$ the true partner from training data, and $G(x)$ the generated candidate:

$${\mathcal{L}}_{\text{cGAN}}(G,D)={\mathbb{E}}_{x,y}[\log D(x,y)]+{\mathbb{E}}_x[\log (1-D(x,G(x)))]$$

Maximize it over $D$, minimize over $G$ — the same two-sided game as before, except the logs now score pairs rather than single images. On a fake pair, $D$ outputs a small number, so $1-D(\cdot )$ grows toward one and the second logarithm turns large-negative, punishing the generator through minimization; on a genuine pair the first logarithm rewards the discriminator for recognizing the true match.

Adversarial pressure alone leaves room for drift: the output could be realistic-looking yet structurally off from the input — a plausible bottle standing at a different angle. So a reconstruction term joins the objective:

$$G^{\ast }=\arg \underset{G}{\min }\underset{D}{\max }{\mathcal{L}}_{\text{cGAN}}(G,D)+\lambda {\mathbb{E}}_{x,y}[\Vert y-G(x){\Vert }_1]$$

Here $G^{\ast }$ denotes the best generator found by solving the two-sided problem, and $\lambda$ balances realism against faithfulness. The L1 norm — the sum of absolute pixel differences — pulls $G(x)$ toward $y$ pixel by pixel.

12.8.4 The Patch Discriminator

Whole-image judgment is wasteful and coarse. The Pix2Pix discriminator instead breaks both images into smaller pieces and compares patch against patch — this piece of the generated image against the aligned piece of the true one, and so on across all overlapping windows — penalizing each patch in the output that looks fake. This is the PatchGAN design, often just called a patch-wise discriminator.

12.8.5 Results and Failure Modes

The gallery of successes: outlines turning into realistic color images, food and dessert photographs synthesized from sketches, a crisp button-slide image colored convincingly, sneakers rendered from their edges. Then the cautionary exhibits.

Real-world: the English-to-German framing above is the same shape as neural machine translation, and sketch-to-product rendering — the sneaker case — is a direct e-commerce workflow where shoppers or designers draft products by drawing outlines.

12.9 Auxiliary Classifier GAN (ACGAN)

ACGAN sits in the conditional family alongside Pix2Pix-style conditioning, with one twist: the discriminator wears two hats.

12.9.1 Class-Conditioned Real-Fake Scores

Beyond declaring an image real or fake, the ACGAN judge also predicts the class label. One network, two output jobs:

Its discriminative loss distinguishes between real and fake examples conditioned on their respective class labels — a real-sample loss and a fake-sample loss, each computed with the label attached, combine into the training signal. The classification branch adds its own demand: when the judge guesses the wrong class for an image it believes is real, it pays, and that pressure travels back into the generator as "make samples not just realistic but recognizable". On the generation side the goal stays the familiar conditional one from Section 12.7: produce images that belong to a requested particular class.

Compared with plain CGAN, the division of labor differs. CGAN shows the label to the judge as extra input information; ACGAN makes the judge recover the label from pixels alone, which sharpens class separation inside the learned features. Know the architecture split — generation branch plus classification branch inside the judge — and the label-conditioned loss structure, and you have this variant's essence.

12.10 CycleGAN

Pix2Pix has an appetite that is hard to feed: paired data. CycleGAN drops that requirement and learns mappings between two image collections that share a concept but have no point-to-point alignment.

12.10.1 Learning from Unpaired Sets

Paired training, as Pix2Pix needs it, means for every outline boot a matching photo of that same boot, dress shoes with dress shoes, sneakers with sneakers — correspondence pixel by pixel, same background included. Such pairs are difficult to acquire. CycleGAN accepts two unrelated piles: lots of photos, lots of paintings, nothing aligned — an unpaired setup.

The classroom moment that planted the idea went like this:

That observation is the entire task specification: learn the transformation that takes an original photo and renders it as a painting — the examples shown belonged to the oil-painting class, in the style of Monet — or runs the other way, taking something that looks like a painted canvas and making it look like a mobile-phone capture. Feed a Van Gogh-style image and receive a Monet-style image out. The correspondence is abstract: the model captures the concept of painting versus realism, not a pixel map. No teacher ever says which painted stroke matches which photo pixel; the shared idea of "style" is all that links the piles.

The horse-and-zebra demo makes it concrete: train on lots of arbitrary horse images and lots of arbitrary zebra images — multiple zebras welcome, no shared backgrounds — and the trained system paints zebra stripes onto a given horse, producing a zebra image that exists nowhere in training. Other mappings shown: satellite imagery to street maps, cloudy-day images to bright-day images and back. Results are not perfect — stripe placement occasionally wanders onto sky or grass — but as contextual texture transfer it is striking.

12.10.2 Two Generators, Two Discriminators

One generator cannot police itself, so CycleGAN deploys two generators and two discriminators. Generator $G$ maps domain X to domain Y — horse to zebra. Generator $F$ maps Y back to X — zebra to horse. Discriminator $D_Y$ judges whether an image presented as Y looks like a genuine member of the zebra pile; discriminator $D_X$ does the symmetric duty for horses, separating $F(y)$ from real samples $x$. Each half is a complete adversarial game on its own; the magic lies in forcing the two halves to close a loop.

12.10.3 Forward and Backward Cycle Consistency Loss

Here is the loop. Take a horse $x$, apply $G$, get $\hat{y}$; apply $F$ to $\hat{y}$, and you should land back near $x$. Symbolically $F(G(x))\approx x$. Run it backward too: $G(F(y))\approx y$. The cycle consistency loss measures both round trips:

$${\mathcal{L}}_{\text{cyc}}(G,F)={\mathbb{E}}_{x\sim X}[\Vert F(G(x))-x{\Vert }_1]+{\mathbb{E}}_{y\sim Y}[\Vert G(F(y))-y{\Vert }_1]$$

Here $X$ is the collection of real images (horses) and $Y$ the collection of target-style images (zebras); each expectation averages over its whole collection. The L1 norm is the stated preference over squared error, because it resists noise better — outlier pixels drag an L1 penalty less than they drag a squared penalty.

Without this loss the system would happily learn a mapping and an inverse that agree on nothing: $G$ could shuffle every horse into an arbitrary encoding that $F$ decodes however it likes, and both discriminators would still be satisfied. Round-trip agreement is what forces the learned transformation to preserve content while changing style.

12.10.4 Adversarial Terms and the Full Objective

Each direction also carries a standard adversarial push, written here in least-squares form as presented:

$${\mathbb{E}}_{x\sim X}[(D_Y(G(x))-1{)}^2]$$

Driving $D_Y$'s score on the generated zebra toward one is exactly what drives this term toward zero — proof that a real-looking zebra was manufactured.

Assemble everything, optimizing over generators $G,F$ and judges $D_X,D_Y$:

$$\underset{G,F}{\min }\underset{D_X,D_Y}{\max }{\mathcal{L}}_{\text{adv}}(G,D_Y)+{\mathcal{L}}_{\text{adv}}(F,D_X)+\lambda {\mathcal{L}}_{\text{cyc}}(G,F)$$

One generator-judge pair fights over the X-to-Y direction, the other over Y-to-X, and both round-trip penalties tie the knot. Applications extend past animals and art: camera photos to oil paintings, satellite views to maps, weather shifts between cloudy and bright — anywhere two visual domains relate by style or sensor rather than by alignment. Film studios use the same recipe for era conversion of archival footage, and autonomous-driving teams convert sunny recorded drives into rainy ones to widen test coverage without new data collection.

Exam Guidance Summary

• Past question papers for the regular exam are being released; use them for practice.
• Expect comprehensive-exam questions drawn directly from the assignments, both the current one and the earlier one — doing the group work diligently is exam preparation, and this deliberately raises the weight on experiential learning. Treat every assignment run you complete as one fewer surprise on exam day.
• The assignment spans implementing VAEs at different beta values (an ordinary VAE is the special case beta equals one), VQ-VAE at different codebook sizes, and WGAN versus WGAN-GP — understand the pros and cons of each from running them, not just from reading about them.
• Marks are reserved for achieving expected evaluation quality: know the typical FID and PSNR ranges these techniques reach on CIFAR-10-like data, and treat falling-short metrics as a signal to retune architectures rather than to accept. Code-generation tools can write code, but they will not hand you results commensurate with the techniques — the results come from tuning and debugging.
• Experiment with the weight clipping parameter $c$ in the WGAN part to reach reasonably low FID scores; no value is prescribed on purpose, because hyperparameter search is itself the skill being trained.
• Implement the WGAN-GP training loop exactly as specified: five critic iterations per generator update, Adam optimizer, normal Gaussian latents, and no batch normalization in the critic.
• Pretrained models and transfer learning are permitted in the assignment — smart reuse is a practical technique, especially where teams do not build such products from scratch. FID computation uses pretrained Keras InceptionV3 weights, so learn how that scoring pipeline is wired.
• Know the power-method route to spectral norms, the hinge-loss value walkthrough, and the paired-versus-unpaired distinction between Pix2Pix and CycleGAN — these three items compress most of this lecture's testable detail into a short revision list.

Key Industry Applications

• Hyperparameter fluency: choosing a technique is table stakes; producing results with it — by tuning knobs like the clip constant or patch size — mirrors day-to-day industry expectations, where the deliverable is a working metric, not a working sketch.
• Reproducibility engineering: generative systems are stochastic, so industrial deployments wrap them in fixed seeds, pinned configurations, and evaluation harnesses to show dependable behavior on demand — the same reason chat assistants give slightly different answers to identical prompts.
• Pretrained-model economics: organizations that do not specialize in building such products routinely start from pretrained networks and adapt them with limited proprietary data via transfer-learning principles, trading training cost for fine-tuning effort.
• Synthetic data creation: SAGAN-class models generate full-image samples resembling a large curated corpus while remaining distinct from it — useful wherever new training data is needed but collection is costly or privacy-sensitive.
• Vision-to-vision workflows: grayscale colorization, sketch-to-product rendering, satellite-to-map generation, and photo-to-painting styling all follow the Pix2Pix or CycleGAN recipes covered here; e-commerce catalogs and media studios run production variants of exactly these pipelines.
• Evaluation practice: FID scoring with InceptionV3 embeddings is the standard yardstick for comparing GAN variants in production experiments, which is why fluency in reading (and distrusting) FID tables transfers across teams.