From CycleGAN to Diffusion Models

These notes close out the GAN family and open the door to diffusion models. We first recap CycleGAN, its losses, and where it breaks. Then come two high-end generators, BigGAN and the StyleGAN family. The second half builds the core idea of diffusion models: destroy data with noise step by step, then learn to undo it.

13.1 CycleGAN Recap and Its Limits

13.1.1 What CycleGAN Transfers, and Where Else It Shows Up

Recall the motivating picture from last time: you have a set of images with horses and a set of images with zebras. The goal of CycleGAN (a GAN that learns translations between two image collections) is to learn a transfer that makes a horse look like a zebra, and one that goes back the other way. The interesting part is that stripes are hard to pin down quantitatively. You cannot say "add stripe at row 40, column 12." Stripes are abstract characteristics of the image style - a holistic look rather than a pixel recipe - yet the network learns to move them across.

The horse-and-zebra pair is only the teaching example, chosen because everyone can instantly see what "characteristics" means here. Think of it like an accent: you cannot list the physical movements of a person's tongue that make a Scottish accent, but you recognize one immediately and could ask an actor to adopt it. CycleGAN does the same job for images. Like an accent, though, the characteristic rides on top of the content - which is exactly where this method will later break down (an accent cannot turn a sentence into a different sentence, and CycleGAN cannot turn a horse into a cat).

The same machinery handles many other swaps of style:

• Text: take an informal passage and make it formal, or take a friendly tone and rewrite it as authoritative. This is characteristic transfer for language rather than pixels.
• Art: move between shape-driven modern art and the soft Impressionist manner of painters like Monet. You could add painterly characteristics to geometric-looking pictures or the reverse. Many photos and many Monet paintings exist, but no photo has a Monet "twin" - which is precisely the situation CycleGAN is built for.

So the general skill is: transfer qualities that resist quantitative description from one collection of data to another.

13.1.2 CycleGAN versus Pix2Pix: Paired versus Unpaired Data

It helps to separate CycleGAN from a cousin we saw earlier, the pix2pix GAN (a translation model trained on matched before/after pairs).

In pix2pix you train on paired images: the very same scene given to you twice. A grayscale photo together with the color photo of that same scene is the classic case. Another example: a picture of an animal together with a second image holding just its silhouette or edge map. Each training item is a matched couple, which makes the pairing requirement quite restrictive - somebody had to build those aligned pairs first, and building them is expensive manual labor.

CycleGAN drops that restriction. You only need two buckets of data - horses in one, zebras in the other - with no one-to-one alignment at all. That unpaired setting is what makes CycleGAN so versatile: any two collections of images (or texts) with different styles become usable training material.

When to pick which: if true aligned pairs exist (or can be made), pix2pix gives tighter control; if only two unrelated collections exist, CycleGAN is the option.

13.1.3 Two Generators, Two Critics

The architecture reflects the round trip. There are two generators:

• $G$ maps from domain $X$ to domain $Y$. With our example, $G$ takes a horse image $x$ and produces a zebra-looking image.
• $F$ maps back from $Y$ to $X$, turning a generated zebra into something horse-like again.

There are also two discriminators, sometimes called critics:

• $D_Y$ judges whether an image is a real zebra from the zebra collection.
• $D_X$ judges whether an image is a real horse.

Picture two parallel conveyor belts running in opposite directions between two warehouses. Each belt has a quality inspector at its end who only knows what genuine goods from their own warehouse look like. This doubled setup - a pair of generators plus a pair of critics - is the fundamental structural difference between CycleGAN and the single-generator GANs we studied before it. Every piece of the loss machinery below belongs to one of these four players.

13.1.4 The Adversarial Loss Term

Each direction gets its own adversarial loss. For the horse-to-zebra side, the loss described in class was "maximise log of $D_Y(y)$ plus log of one minus $D_Y(G(x))$". Written out for samples $x\sim p_{\text{data}}(x)$ drawn from the horses and $y\sim p_{\text{data}}(y)$ drawn from the zebras:

$${\mathcal{L}}_{\text{GAN}}(G,D_Y,X,Y)={\mathbb{E}}_{y\sim p_{\text{data}}(y)}[\log D_Y(y)]+{\mathbb{E}}_{x\sim p_{\text{data}}(x)}\left[\log \right(1-D_Y(G(x))\left)\right]$$

Every symbol, named:

• $D_Y(\cdot )$ outputs a realness score in $[0,1]$ - how confident the critic is that its input is a genuine zebra.
• $G(x)$ is the fake zebra created from horse $x$.
• $p_{\text{data}}(\cdot )$ is the empirical distribution of the dataset - averaging over it (the $\mathbb{E}[\cdot ]$ operator) means "average over many sampled images."
• $X$ and $Y$ name the two domains (horses and zebras).

The push-pull works like this. Notice first that because $D_Y\in [0,1]$, both logarithms are at most zero: $\log (1)=0$ is the best possible value, and values fall toward negative infinity as the argument approaches zero.

• For a real zebra $y$, we want a high value of $D_Y(y)$, so $\log D_Y(y)$ is close to zero - its best case.
• For a fake zebra $G(x)$, the critic should output a small score, pushing $\log (1-D_Y(G(x)))$ toward $\log (1)=0$. Log of a small number is a strongly negative quantity, so this term rewards pushing $D_Y(G(x))$ toward zero.

We maximize the whole expression with respect to $D_Y$ and minimize it with respect to $G$. Minimizing means $G$ gets better and better until $D_Y(G(x))$ approaches one - the critic itself starts believing the fake zebras. When training succeeds, a horse is converted into a zebra so convincing that it looks like the true zebras sitting in the training set.

13.1.5 The Cycle Consistency Loss

Adversarial pressure alone says "look like a zebra" but says nothing about preserving the horse underneath. A critic cannot see the original horse, so nothing stops $G$ from inventing a completely different zebra-scene. So CycleGAN adds a round-trip demand. Take the generated zebra $G(x)$, pass it through the backward generator $F$, and compare the result with the original horse $x$. The comparison uses the L1 norm - the sum of absolute pixel differences, written $\Vert \cdot {\Vert }_1$:

$${\mathcal{L}}_{\text{cyc}}(G,F)={\mathbb{E}}_{x\sim p_{\text{data}}(x)}[\Vert F(G(x))-x{\Vert }_1]+{\mathbb{E}}_{y\sim p_{\text{data}}(y)}[\Vert G(F(y))-y{\Vert }_1]$$

Walk through both halves:

1. First half: horse to zebra (via $G$), then zebra back to horse (via $F$). The returned horse must nearly equal the starting horse, measured by the L1 norm.
2. Second half: zebra to horse (via $F$), then horse back to zebra (via $G$). That loop must return the original zebra too.

Both directions must stay consistent, which is exactly why the method is called cycle consistency. The round trip acts as a virtual supervisor: even without a single matched pair, the demand "go there and come back unchanged" pins the translation down.

Why L1 rather than the squaring L2 norm? Absolute differences penalize every wrong pixel proportionally and tolerate a few extreme outliers, which discourages the blur that squared-error averages tend to produce on images. Texts on image translation routinely prefer L1 for exactly this reason.

Exam note: be ready to state why there are two L1 terms and which generator pair each one supervises - the first supervises the $G$ then $F$ ordering starting from horses, the second the mirrored $F$ then $G$ ordering starting from zebras.

13.1.6 Full Objective, Minimax, and Patch-wise Scoring

Put the pieces together. The full objective collects the adversarial term for $G$ against $D_Y$ (which drives high-quality zebra creation from horses), the mirrored adversarial term for $F$ against $D_X$ (zebra-to-horse generation judged by the real-horse critic), and the cycle terms:

$$\underset{G,F}{\min }\underset{D_X,D_Y}{\max }{\mathcal{L}}_{\text{GAN}}(G,D_Y,X,Y)+{\mathcal{L}}_{\text{GAN}}(F,D_X,Y,X)+\lambda ({\mathcal{L}}_{\text{cyc}}(G,F)+{\mathcal{L}}_{\text{cyc}}(F,G))$$

Here $\lambda$ (lambda) balances how strongly the round-trip constraint pulls against realism. In the discussion it was left implicit, but standard presentations include it explicitly - the original CycleGAN work sets $\lambda =10$, deliberately letting the consistency demand dominate so that content survives translation. We maximize this objective with respect to both discriminators and minimize it with respect to both generators - the same minimax game we solved for earlier GANs, now with four players instead of two.

One more practical detail: the discrimination is patch-wise. Instead of scoring every pixel jointly into a single real/fake verdict, the critic breaks the image into patches and evaluates realness patch by patch, averaging those local judgments. Concretely, the critic is a purely convolutional classifier whose every output unit looks at a small receptive field and votes on whether that region looks real. Local texture is what gives fakes away, so judging locally sharpens the learning signal. This is the same trick used in pix2pix, reused here for both critics.

Example generator and discriminator architectures were shared through the course portal if you want concrete layer-by-layer blueprints.

13.1.7 Where CycleGAN Breaks

CycleGAN has real limitations, and seeing them is instructive.

Second, some tasks need geometric change, and CycleGAN cannot handle it. Think about turning a dog into a cat: you are not just recoloring fur, you are reshaping the structure of the face - muzzle length, ear placement, eye size. As you try to impose cat characteristics, the object's geometry would have to deform, and the cycle-consistency machinery does not support that kind of structural edit. Characteristic-swap tasks suit CycleGAN; shape-reshaping tasks do not.

Two beginner traps worth flagging before you walk into an exam:

1. Assuming CycleGAN understands objects. It matches global statistics between domains; it has no notion of "horse" or "person" as things. Any segmentation-like behavior you observe is an accident of the data, not a capability.
2. Reaching for CycleGAN when pairs exist. With true paired data, pix2pix usually gives cleaner, more faithful results - CycleGAN's freedom becomes a liability, not a feature.

Real-world: this limitation profile matters wherever style migration meets people - photo editing apps translating portraits must either segment the human first or accept artifact bleed onto skin, hair, and clothing. The broader field lesson is that unsupervised translation trades supervision for a consistency assumption, and every assumption draws a boundary around the method.

13.2 BigGAN: Scale, Fidelity, and Control Knobs

13.2.1 Why "Big", and How Deep

BigGAN earns its name from its compute appetite. It generates very large images - on the order of 1024 by 1024 - while keeping crisp, fine detail. Large size alone is easy; the trick is doing it without blocky, low-resolution artifacts. BigGAN delivers high resolution and high fidelity (fine detail actually rendered, not smeared) together. At 128 by 128, a face may have no visible eyelashes or skin pores; at 1024 by 1024 those details exist and look right.

How does it get there? By stacking layers aggressively. The generator starts from a tiny spatial block produced from the random vector - think 4 by 4 - and climbs a long ladder:

Both networks are very large. A caution from class: you will not retrain BigGAN-scale models on the resources allotted in this course - treat it as an awareness-level architecture, with references provided for when a real need arises.

13.2.2 Stabilizers: Residual Blocks, Self-attention, Orthogonal Weights

Training networks this deep runs straight into vanishing and exploding gradients (gradients that shrink to nothing or blow up as they travel through many layers). So BigGAN borrows the standard remedies and adds its own:

• Residual blocks with skip connections. A residual block computes a small change and adds it back to the input: output equals input plus f(input). The skip connection is that identity path which carries the input around the transformation untouched. Gradients can then flow backward through the shortcut instead of squeezing through every layer, which keeps them healthy across sixteen-plus layers.
• Self-attention, the mechanism introduced in the SAGAN architecture, lets each output location decide where to draw information from when rendering a pixel. Attention tells the layer which regions matter most for the pixel being produced - so distant, coordinated parts of an image (an ear on one side, an eye on the other) can agree instead of being drawn independently.
• Orthogonal regularization: every layer of the generator holds a bundle of weight vectors. BigGAN adds a penalty encouraging these vectors to stay mutually orthogonal (at right angles to each other):

$$\underset{W}{\min }\sum _{p\ne q}|w_p^{\top }{w}_q{|}^2$$

Here $W$ collects the weight vectors $w_p$ of a layer, and $w_p^{\top }{w}_q$ is their inner product - zero exactly when two vectors are orthogonal. Squaring and summing over all unequal pairs $p\ne q$ punishes any overlap between directions. Why bother? Orthogonal directions carry independent, non-overlapping information about the input, so each weight vector captures distinct, canonical factors rather than redundant copies of each other. It doubles as a regularizer that keeps the huge model from overfitting.

13.2.3 The Truncation Trick: Fidelity versus Variety

Here is one of the most exam-worthy ideas in BigGAN. The input to the generator is a random vector $z$, drawn dimension by dimension from a Gaussian - a bell-shaped probability distribution centered on the mean. Sampling from the full bell curve sometimes lands you far out in the low-probability tails, and those rare far-out vectors produce strange, defective images.

The BigGAN inventors added a truncation trick: generate the random number, but keep it only if it falls near the mean, inside a high-probability band. Formally, restrict each component $z_i$ of the latent vector to a window of radius $\psi$ (psi, measured in standard deviations) around the center:

$$z_i\in [-\psi ,+\psi ],z\sim \mathcal{N}(0,I)\text{kept only inside the band}[-\psi ,+\psi ]$$

In practice this is implemented by rejection sampling: draw a candidate $z$, measure how far it sits from the mean in standard deviations, and redraw it if it exceeds $\psi$. Keeping the latent inside the high-probability range made the generated images noticeably clearer and higher in quality - even at 1024 by 1024.

But there is a price, and it was posed to the room as a question:

Picture the bell curve with a shaded central band. Narrow the band and the shaded spike grows taller while its footprint shrinks - sharper samples, fewer distinct ones. Widen it toward the full curve and the opposite happens.

13.2.4 Conditional Generation: Class Information at Every Layer

BigGAN is a conditional GAN: besides the random vector, you specify a target class - dogs, outdoor scenes, even X-ray images (the same conditioning idea applies across domains). The class label passes through an embedding layer (a lookup table turning each label into a learned numeric vector), and the resulting embedding is injected into the generator and also into the discriminator, so the critic knows which class it should be judging.

The subtle part: class information is fed at every layer of the generator, not just at the input. If you inject it only at the first layer, the class signal fades away as activations pass through the many subsequent layers - like whispering instructions once at the start of an assembly line and hoping the last station still remembers them. Feeding it into every block keeps the conditioning alive all the way to the output.

13.2.5 Splitting the Latent Vector: Layer Specialization

The second structural innovation concerns how $z$ enters the network. Rather than handing the whole vector to the first layer, BigGAN splits $z$ into chunks - think $z_1,z_2,z_3,z_4,z_5$ - and sends different chunks to different layers of the generator.

Why bother? Feeding the entire vector only at the entrance entangles all the characteristics of the image into one mixed signal. Splitting lets different layers specialize: one subset of values steers texture types, another shapes, another controls how much a face smiles, another whether eyeglasses appear, another skin tone. Each layer becomes responsible for particular characteristics because its own chunk of randomness controls them.

If this rings a bell, it should. We met the same disease and cure in VAEs. Recall β-VAE: with beta equal to one you have a standard VAE whose latent space merely matches a Gaussian prior. Raising beta pushes the latent space to orient along the axes, making the covariance matrix diagonal instead of full of cross-terms - disentangled factors, meaning each coordinate controls one human-interpretable attribute. BigGAN achieves a similar disentangling effect architecturally: chunking the latent and routing chunks to dedicated layers separates the factors that previously got mixed.

13.2.6 Architecture Sketches, Huge Batches, and the SNGAN Link

A concrete sketch ties it together. In the walkthrough diagram, the raw latent was shown as a 160-dimensional vector, while earlier narration described a 128-dimensional $z$; published BigGAN models likewise use a 128-dimensional $z$, so treat the exact figure as a course-diagram detail rather than a fixed constant. The number of classes was stated as 128, encoded through a learned embedding. Latent plus embedding together form the generator's input. Resolutions then climb stage by stage while channel counts shrink. The generator ends with a tanh hyperbolic activation, squashing outputs into a fixed range so they can represent an image. The discriminator runs the mirror-image pipeline, shrinking spatial size while growing channels, and it consumes the class information too.

Two more practical notes. First, BigGAN tolerates enormous batch sizes, and quality holds up - batch size should scale roughly in proportion to the total number of training samples. Second, model size can be pushed arbitrarily; quality keeps improving with scale.

On the family tree: BigGAN subsumes tricks from its predecessors. It incorporates spectral normalization (the stabilization device introduced in SNGAN, which rescales weight matrices so the discriminator cannot grow arbitrarily sharp), alongside self-attention from SAGAN, orthogonal regularization, per-layer conditioning, latent chunking, and truncation. Stack all of that and it outperforms techniques like SNGAN by a wide margin - at a matching compute cost.

Real-world: BigGAN-class conditional synthesis is the ancestor of today's high-resolution stock-image generators; the control knobs you just met (class conditioning, truncation strength) survive in modern production systems as the levers operators pull when outputs need to be sharper or more varied. For the exam, the architecture itself is awareness-level - the truncation trade-off is the part to know cold.

13.3 StyleGAN and Its Three Versions

13.3.1 Styles, and the Mapping Network from z to w

What is "style"? Nobody can define it precisely, yet everyone sees it instantly - a person looking old-fashioned versus modern, formal versus casual, stern versus warm. StyleGAN's job is to grab those elusive style attributes and control them, while producing extremely realistic faces.

Its key move reshapes what we did with the latent vector. In plain GANs, $z$ goes straight in. In BigGAN, $z$ got split into chunks. StyleGAN does neither: it passes $z$ through a mapping network built from 8 fully connected layers, producing a new vector $w$. Training shapes $w$ so it aligns with the styles present in the training data - each coordinate of $w$ drifts toward representing one human-meaningful factor rather than an entangled mixture. Then, at every block of the generator, a learned affine transformation $A$ - a linear combination map of $w$ - converts $w$ into the style parameters that block will use.

Think of $z$ as raw pigment squeezed straight onto a canvas, and $w$ as a palette that has been organized first: same randomness, but arranged so a painter can reach for "warmer skin tone" or "grayer hair" deliberately. The analogy breaks where palettes are fixed by the manufacturer - here both the palette organization ($w$) and the mixing recipe ($A$) are learned.

So the pipeline becomes:

$$z⟶\text{mapping network (8 fully connected layers)}⟶w⟶A(w)\text{per block}⟶\text{style parameters steering generation}$$

The network transforms raw randomness into style coordinates before touching the image.

13.3.2 AdaIN: Adaptive Instance Normalization

How do style parameters actually steer a block? Through adaptive instance normalization, or AdaIN. The formula discussed in class was read out as "x-i minus mu of x-i, divided by sigma of x-i, multiplied by y-s-i and y-b-i":

$$\text{AdaIN}(x_i,y)=y_{s,i}\frac{x_i-\mu (x_i)}{\sigma (x_i)}+y_{b,i}$$

Read it piece by piece:

• $x_i$ is channel $i$ of the block's feature map.
• $\mu (x_i)$ and $\sigma (x_i)$ are that channel's mean and standard deviation computed per instance - per single sample being generated, not across a batch like batch normalization.
• The fraction $\frac{x_i-\mu (x_i)}{\sigma (x_i)}$ strips the channel down to zero mean and unit spread - wiping out its current contrast-and-brightness fingerprint.
• $y_{s,i}$ rescales and $y_{b,i}$ shifts the normalized channel; these are the learned style scale and shift derived from the affine transform of $w$.

In one sentence: erase the channel's own statistics, then paint on the statistics the current style asks for. Before each convolution, the block normalizes its input and immediately restyles it this way. Every level of the generator gets its own styles, which is what makes fine stylistic steering possible - coarse blocks receive styles controlling pose and face shape, fine blocks receive styles controlling hair strands and freckles.

13.3.3 Noise Injection for Stochastic Fine Texture

StyleGAN adds one more input: noise, injected directly into blocks at multiple levels. This noise supplies stochastic variation - the tiny random details real photos have. Compare renders with and without it: without noise, surfaces look flat, like airbrushed regions with unnaturally smooth "air" distribution; with noise, you get individual hair strands, pores, grain - genuine fine-grained texture. More noise injection means finer stochastic detail in the rendered result.

Why a separate noise path at all? Because randomness that should move whole styles (age, pose) and randomness that should scatter tiny details (exact positions of stubble hairs) are different kinds of variation. Giving each its own input lets the network keep them apart instead of entangling them in $z$.

Real-world: the classic style-mixing demonstrations let you copy attributes between two synthesized people - take the hairstyle from source A and place it on the identity from source B, swap facial hair distributions, or exchange eyeglass styles. Each attribute lives in specific layers, so mixing at chosen depths moves exactly those characteristics.

13.3.4 Droplet Artifacts and the StyleGAN2 Remedy

Version 1 had a visible flaw: droplet artifacts. Blob-like marks resembling water droplets appear in flat regions of the image, seemingly out of nowhere. The culprit is AdaIN itself - the instance-level adaptive normalization leaves those traces, especially where the image is smooth.

This matters beyond aesthetics. Machine-generated faces became hard to distinguish by eye, so websites appeared that test whether a face photo is artificially generated - a useful reality check now that fakes look authentic unless you know the technology's tell-tale signs.

StyleGAN2 fixed the droplets by removing AdaIN. In class this was described as replacing version 1's mean-and-standard-deviation computation with a modified calculation of the statistics; standard accounts describe the same redesign more precisely as weight demodulation, where the normalization moves from the activations into scaled-down convolution weights whose per-channel gain is divided out after the convolution. Both descriptions point at the same fix: drop the per-instance normalization block, keep learned per-channel scaling. Droplet artifacts disappear once that normalization block is gone.

But StyleGAN2 introduces its own problem: phase artifacts.

13.3.5 Phase Artifacts, Texture Sticking, and StyleGAN3

Phase artifacts are hard to explain in words but obvious in pictures. Consider a synthesized smile: the teeth show left-right symmetry, and the tip of the nose sits centered above them. Now rotate or shift the face in the image. The head moves - but the midpoint of the teeth stays put. One part of the image lags while another swings away, a clear phase mismatch between image regions. Features literally stick in place. This failure is called texture sticking, and the normalization design of StyleGAN2 is again the responsible block.

StyleGAN3 attacks it with a more complex architecture: extra layers in the generator working alongside the mapping network, plus query-based features for sampling detail at arbitrary positions - the phrase in class was partly garbled, but the standard account is a redesigned generator that treats the image as a continuous signal rather than a grid of pixels, so textures ride along when the content rotates or translates. Texture sticking goes away. The cost: StyleGAN3 is the most expensive of the three to run, and it also produces the best results.

The version story in one line: StyleGAN1 has droplets from AdaIN; StyleGAN2 removes AdaIN, killing droplets but adding texture sticking; StyleGAN3 kills sticking with more machinery and tops the quality charts.

Extra reading lists further GAN variants; more pointers were promised for the course portal.

13.4 Diffusion Models: The Core Idea

13.4.1 Ink in Water: The Physical Picture

Post-GAN, the technique making the biggest impact on high-quality generation is the diffusion model.

Two properties of the ink make it the perfect anchor for what follows. First, the spreading happens in many tiny, almost random nudges - not one big jump. Second, the end state carries no trace of where the drop landed: uniform pale water looks the same no matter which corner you dripped into. Diffusion models copy both properties exactly.

13.4.2 From Physics to Data: Noise Up, Then Learn to Undo

Apply the same idea to data. Start with real data - say a flower image. Add a little noise. Add some more. Keep going step by step, $n_1,n_2,n_3$, iteratively. If you iterate long enough, the original character disappears completely and what remains looks like pure random data - asymptotically something like a Gaussian distribution, just like the pale uniform water at the end of the ink experiment.

Generation reverses the arrow, and this is the heart of the idea:

How is the reversal learned? Through likelihood maximization: choose the reversal-network parameters so that the data regenerated along the reversed chain has maximal probability under the model. Section 13.4.9 makes this concrete.

Speed warning: iterative sampling makes generation slow - you create the image gradually, step after step, often hundreds or thousands of them. And the idea reaches beyond images: diffusion can serve natural language processing too, provided you move from a continuous distribution to a discrete one; latent diffusion models exist for text.

13.4.3 Closest Relatives: Variational Autoencoders and Flow Models

Which familiar model is closest? The variational autoencoder. In a VAE you sample from the latent space's Gaussian distribution and push the sample through the decoder once to get data. Training maximizes data likelihood minus the KL divergence between the encoder's output distribution and the standard Gaussian (KL divergence measures how one probability distribution differs from another). Diffusion differs in one key way: the VAE decodes in a single step, while diffusion walks through many steps.

The truer relative is the hierarchical VAE. There you chain latents - $z_1$, then $z_2$ from it, then $z_3$ - with layered encoders and layered decoders unwinding them back to data. Those stacked decoding stages play the role of diffusion's repeated denoising iterations. A useful way to see diffusion: a hierarchical VAE whose encoder is fixed by design (it just adds known noise), so only the decoder chain needs learning.

Flow models share the skeleton too. A flow model applies bijective transformations - invertible maps where every input has exactly one output and back again - to warp the data distribution into a simple Gaussian, samples from that Gaussian, pushes the sample back through inverse transformations, and trains by maximizing likelihood. Same recipe as diffusion: sample simple noise, apply learned reversible steps, maximize likelihood. Diffusion belongs to this family in spirit; it just replaces clean invertible maps with learned incremental denoising.

13.4.4 Text-to-Image: What Kind of Learning Is This?

Modern diffusion systems generate photorealistic scenes from typed prompts - "a robot cooking dinner in the kitchen", or "a bread, an apple, and a knife on a table". Where does the knowledge of what a bread or apple looks like come from? Several students pressed on exactly this point:

There is also a self-training loop that sharpens prompt-matching. Given a generated image, run a captioner - a CNN followed by attention and an LSTM/RNN decoder - to describe the picture in words. Compare that machine-written caption with the input caption and train the generator to reduce the difference. Round after round, the system learns to emit images whose captions match their prompts. The same pattern extends to video: analyze the prompt, synthesize successive frames, use an image-to-next-frame transformation, and retrain so consecutive frames match the requested scene.

13.4.5 Missing Categories, Bias, and Hallucination

What happens when the prompt names something absent from training? This drew another cluster of questions:

This leads to a law worth remembering.

13.4.6 A Quick Timeline of Generative Quality

Where do diffusion models sit historically?

• 2013 - original VAE: reasonable results on small images, with visible artifacts. (Not hierarchical VAEs or VQ-VAEs - the plain original.)
• 2014 - original GAN: worked on small black-and-white images in the CIFAR-10 era. A fascinating technique from the minimax-optimization standpoint: rather than learning the probability distribution directly, it learns it implicitly through the two-player game.
• 2016 - PixelCNN and its variants: autoregressive generation improves (the model writes the image pixel by pixel, each pixel conditioned on the previous ones).
• 2019 - BigGAN: sharp, large samples - the architecture we met in Section 13.2.
• After that - diffusion models: the biggest jump in quality, including systems that start from a textual description and render realistic scenes.

One caveat stands: generation stays slow, because quality emerges through many iterative refinement steps.

13.4.7 Forward Process Mathematics

Now the math. Notation first: $x_0$ is a sample from the training data distribution - randomly drawn from the dataset. $t$ indexes the noise step, running up to a final time $T$, typically a reasonably large number like 1000. At $t=T$ the image has become $x_T$, pure Gaussian noise. $I$ denotes the identity matrix, giving independent noise in every dimension.

Each forward step adds zero-mean Gaussian noise whose variance grows with $t$. Written as a conditional distribution (shown in class for the first step):

$$q(x_1\mid x_0)=\mathcal{N}\left(x_1;\sqrt{1-{\beta }_0}{x}_0,{\beta }_{0}I\right)$$

In words: the distribution of $x_1$ given $x_0$ is Gaussian with mean $\sqrt{1-{\beta }_0}{x}_0$ - the previous image shrunk slightly - and covariance ${\beta }_{0}I$. One clarification the class discussion left open: because ${\beta }_0$ sits in the covariance slot, it plays the role of a variance, not a standard deviation; the standard treatment likewise treats ${\beta }_t$ as the variance of the injected noise at step $t$. Equivalently, you scale $x_0$ by $\sqrt{1-{\beta }_0}$ and add fresh noise ${ϵ}_0\sim \mathcal{N}(0,I)$:

$$x_1=\sqrt{1-{\beta }_0}{x}_0+\sqrt{{\beta }_0}{ϵ}_0$$

Then repeat with ${\beta }_1$: scale by $\sqrt{1-{\beta }_1}$, add new noise, and so on.

The schedule matters. Use a small ${\beta }_0$ at the start and increase the betas as time progresses - gentle perturbations early, heavy corruption late. Add noise enough times and the structure of the object in the image is completely destroyed, leaving standardized Gaussian noise.

Two structural facts. First, this is a Markov chain: the noise added at step $t$ depends only on the state at $t-1$ - "the noise here depends on what is here," nothing earlier. Second, training must learn the reverse conditionals: given $x_T$, find $p(x_{T-1}\mid x_T)$; given $x_{T-1}$, find $p(x_{T-2}\mid x_{T-1})$; all the way down to $p(x_0\mid x_1)$.

13.4.8 One Closed-Form Jump via Reparameterization

Something convenient hides in the forward process: you can express the state at any time directly in terms of $x_0$, skipping the intermediate steps. Introduce the reparameterization ${\alpha }_t=1-{\beta }_t$, so each step becomes:

$$x_t=\sqrt{{\alpha }_t}{x}_{t-1}+\sqrt{1-{\alpha }_t}{ϵ}_{t-1},{ϵ}_{t-1}\sim \mathcal{N}(0,I)$$

Here $ϵ$ is a fresh standard Gaussian drawn at every call - invoke the random number generator anew for each image and each step, so every sample receives different noise.

Now push symbols through the recursion until the chain collapses into one closed form. Start by substituting the formula for $x_1$ into the formula for $x_2$:

$$\begin{array}{rlr}{x}_2& =\sqrt{{\alpha }_2}{x}_1+\sqrt{1-{\alpha }_2}{ϵ}_1& \\ & =\sqrt{{\alpha }_2}\left(\sqrt{{\alpha }_1}{x}_0+\sqrt{1-{\alpha }_1}{ϵ}_0\right)+\sqrt{1-{\alpha }_2}{ϵ}_1& \text{(substitute the }{x}_1\text{ formula)}\\ & =\sqrt{{\alpha }_1{\alpha }_2}{x}_0+\sqrt{{\alpha }_2(1-{\alpha }_1)}{ϵ}_0+\sqrt{1-{\alpha }_2}{ϵ}_1& \text{(distribute }\sqrt{{\alpha }_2}\text{)}\end{array}$$

The last two terms are independent zero-mean Gaussians, so their sum is Gaussian with variance equal to the sum of variances:

$$\begin{array}{rlr}\mathrm{Var}& ={\alpha }_2(1-{\alpha }_1)+(1-{\alpha }_2)& \\ & ={\alpha }_2-{\alpha }_1{\alpha }_2+1-{\alpha }_2& \text{(expand both products)}\\ & =1-{\alpha }_1{\alpha }_2& \text{(the }{\alpha }_2\text{ terms cancel)}\end{array}$$

So $x_2=\sqrt{{\alpha }_1{\alpha }_2}{x}_0+\sqrt{1-{\alpha }_1{\alpha }_2}ϵ$ - the same shape as one step, with the product ${\alpha }_1{\alpha }_2$ in place of a single $\alpha$. Repeating the substitution folds every further step in the identical way (each new factor multiplies the running product, and the leftover variance shrinks to one minus the product), which proves the general pattern by induction - the whole recursion collapses into one jump.

Define the cumulative product:

$${\bar{\alpha }}_t=\prod _{s=0}^t{\alpha }_s$$

• alpha-zero times alpha-one times alpha-two, onward to alpha-t; note these alphas shrink toward zero since each factor is below one. Then the marginal distribution of $x_t$ given $x_0$ is Gaussian:

$$q(x_t\mid x_0)=\mathcal{N}\left(x_t;\sqrt{{\bar{\alpha }}_t}{x}_0,(1-{\bar{\alpha }}_t)I\right)$$

Equivalently, assemble the noisy image directly: multiply the known image by $\sqrt{{\bar{\alpha }}_t}$, draw one noise vector, scale it by $\sqrt{1-{\bar{\alpha }}_t}$, and add. A note on indices: the bounds $s=0$ follow the class's zero-based beta indexing from $q(x_1\mid x_0)=\mathcal{N}(x_1;\sqrt{1-{\beta }_0}{x}_0,{\beta }_{0}I)$; common treatments index from beta-one (${\bar{\alpha }}_t={\prod }_{s=1}^t(1-{\beta }_s)$), which changes only the labels, not the mathematics.

Why this matters: pick any training image $x_0$ and any target step $t$, draw $ϵ$, and compute $x_t$ in closed form - no loop over intermediate images needed. This is exactly how training creates its corrupted inputs: jump straight to step $t$, then ask the network to predict the noise that was mixed in, learning the removal process from there.

13.4.9 Reverse Process and the Likelihood Objective

The reverse side learns to walk back down. Learn $p_{\theta }(x_{T-1}\mid x_T)$, then $p_{\theta }(x_{T-2}\mid x_{T-1})$, finally $p_{\theta }(x_0\mid x_1)$, where $\theta$ collects the neural network's parameters. Multiply the whole chain of conditionals - a product of probabilities:

$$\prod _{t=1}^T{p}_{\theta }(x_{t-1}\mid x_t)$$

Maximizing the likelihood of real data through this chain is the training objective. Products are awkward to optimize, so take the logarithm, using the identity that the log of a product equals the summation of logs: $\log (a\cdot b)=\log a+\log b$. The log turns the product into a sum, and minimizing the negated sum gives the same result:

$$\underset{\theta }{\max }\sum _{t=1}^T\log p_{\theta }(x_{t-1}\mid x_t)⟺\underset{\theta }{\min }\left(-\sum _{t=1}^T\log p_{\theta }(x_{t-1}\mid x_t)\right)$$

That is the sketch presented in class: multiply the conditionals, take ln, minimize the negative summed log. How each reverse step is parameterized, and the fuller bound behind this objective - together with energy-based models, whose training ideas influence diffusion - arrives next session.

Sampling then works exactly as advertised. Draw $x_T$ from the Gaussian, apply the trained reversal repeatedly, and out comes an image statistically like the training set. Train two diffusion models on different datasets - flowers in one, dogs in the other - and the same kind of starting noise yields a flower from one and a dog from the other. The training data decides what emerges.

13.4.10 Denoising Autoencoder versus Diffusion Model

A sharp question pinned down the boundary with the denoising autoencoder studied before mid-sem:

13.4.11 Different Starting Noise, Different Flowers

One more exchange clarified what sampling variety really means:

Real-world: this noise-to-image pipeline is the engine inside modern text-to-image products and medical-imaging synthesis tools, where a hospital can train a generator on its own scans and then manufacture realistic-but-artificial training examples for rare conditions. The broader field placement: diffusion moved generation from adversarial games (GANs) to likelihood-based denoising, trading sampling speed for training stability and coverage - the defining trade-off of the current era.

Exam Guidance Summary

• Exam note: the circulated sample papers do not include assignment-style questions, but live exams may - so participate actively in the assignment work.
• Assignment calibration: prior published results on CIFAR-10 define what a good result looks like, and grading compares against them. Plan to iterate more than 100 times, using a reasonably large number of epochs; image sizes are small, so single runs are not hours long, but parameter tuning takes real time.
• Practice past sample papers; generating and solving extra practice questions with ChatGPT was endorsed as preparation.
• The quiz opens right after the assignment deadline with a 7-day completion window - revise the material as you answer.
• Team note: the assignment is sized so that multiple contributors are needed to turn in strong work by the fixed deadline.

Concept-level emphasis flagged during discussion, collected in one place for revision:

1. The truncation trick's fidelity-versus-variety trade-off (narrow the band: crisper but more alike).
2. Choosing pix2pix versus CycleGAN for a given dataset type (paired data favors pix2pix; two unrelated buckets force CycleGAN).
3. The denoising-autoencoder-versus-diffusion distinction (known noise/one step/reconstruct versus estimated noise/many steps/generate).
4. The forward-process schedule - small noise early, heavy noise late - and the closed-form marginal $q(x_t\mid x_0)=\mathcal{N}(x_t;\sqrt{{\bar{\alpha }}_t}{x}_0,(1-{\bar{\alpha }}_t)I)$.

Key Industry Applications

• Unpaired style translation between image collections (horse to zebra as the canonical demo): photo studios and dataset builders translate whole collections without paying for matched pairs.
• Text style transfer - informal to formal register, friendly to authoritative tone: writing assistants and customer-service rephrasers apply the same characteristic-moving machinery to language.
• Artistic style migration between genres, e.g., shape-driven modern art versus Impressionist painting in the manner of Monet: creative tools offer one-click genre conversion of user photos.
• Pix2pix paired translation - grayscale-to-RGB colorization, photos and silhouette/edge maps: restoration pipelines for archives, and sketch-to-render tools in design software.
• BigGAN-class systems for 1024 x 1024 high-fidelity synthesis; conditional class control supports domains from animals to outdoor scenes to X-ray imagery: stock-image synthesis and data augmentation where class labels steer output.
• StyleGAN face synthesis with controllable attributes - hairstyle, facial hair, eyeglasses - via style mixing between source identities: avatar creation for games and virtual presenters.
• Detecting synthetic media: websites exist that test whether a face photo is machine-generated, relevant because artifacts like droplets and texture sticking are the tells; each generator generation fixes old artifacts, so detectors must track new ones.
• ImageNet-trained classifiers supply visual knowledge to generative systems through transfer learning: recognition features learned once are reused wherever a generator must know what objects look like.
• Text-conditioned generation pipelines combine contextual text embeddings (beyond GloVe) with diffusion or GAN-like decoders; captioning networks (CNN plus attention plus LSTM/RNN) provide feedback for self-training: this is the architecture pattern behind prompt-to-image products.
• Prompt-driven video synthesis extends frame by frame with image-to-next-frame transformations matched to the prompt: storyboarding and pre-visualization tools generate moving scenes from scripts.
• Pretrained transformer models on the language side power the text understanding in modern generative stacks: every text-to-image system leans on a language model that turns your words into vectors the image side can condition on.