Diffusion Models and Energy-Based Models

Generative models try to learn how training data is distributed so we can draw new samples from that distribution. Think of the training data as a set of footprints on a beach: a generative model studies the footprints until it can produce new footprints indistinguishable from the real ones.

This session finishes the diffusion story started previously and adds its fast variant, DDIM (Denoising Diffusion Implicit Model). It also compares all the generative families seen so far — GANs, VAEs, and diffusion — on speed, diversity, and quality. Then it opens a brand-new family: energy-based models, which define probabilities from first principles using an energy function borrowed from statistical physics.

The road ahead:

1. A quick recap of the forward diffusion process and its closed-form shortcut.
2. The reverse denoising process — what we train, and why it is tractable only with help from the data point itself.
3. The training objective, seen as a hierarchical VAE, collapsing to one beautifully simple loss: predict the noise that was added.
4. The U-Net architecture that plays the role of noise predictor, and how time information enters it.
5. How generation quality is scored (FID and IS).
6. DDIM — same training, faster sampling through data prediction and step-skipping.
7. A side-by-side comparison of VAE, diffusion, and GAN pipelines, including the generative trilemma.
8. Conditional generation (text-to-image) and latent diffusion (Stable Diffusion).
9. Energy-based models: what makes a valid density, how the energy formulation works, why comparing beats evaluating, the Ising model, and product of experts.

14.1 Forward Diffusion Recap

14.1.1 Adding Noise Step by Step

Start from a training sample ${\mathbf{x}}_0$. The forward diffusion process adds a small amount of Gaussian noise again and again: add noise to ${\mathbf{x}}_0$ to get ${\mathbf{x}}_1$, add noise to ${\mathbf{x}}_1$ to get ${\mathbf{x}}_2$, and so on. After enough steps the signal is gone and only noise remains, so the process ends at a standard normal distribution.

The wider goal behind this is generative modeling itself. We want to estimate $p(\mathbf{x})$, the probability of the data. Everything in this course builds different tools for modeling that probability — GANs, VAEs, flows — and diffusion is one more such tool.

Intuition. Picture a drop of ink in a glass of water. Stir gently once: the ink spreads a little but the blob is still visible. Stir again and again: eventually every trace of where the drop landed is gone and the water looks uniformly mixed. Diffusion does exactly this to an image — each stirring step is a small noise addition, and after enough stirs the original picture is unrecoverable. The analogy breaks only in one useful way: unlike ink, we will be able to compute exactly how much signal survives at any step, in closed form.

One noisy step mixes the previous state with fresh noise. Written per step:

$${\mathbf{x}}_t=\sqrt{{\alpha }_t}{\mathbf{x}}_{t-1}+\sqrt{{\beta }_t}{\mathbf{ϵ}}_t,t=1,\dots ,T$$

Here ${\mathbf{ϵ}}_t\sim \mathcal{N}(0,\mathbf{I})$ is the random noise vector drawn at step $t$, with the same shape as the data (for a color image of height $H$ and width $W$, it is a tensor of shape $H\times W\times 3$). The scalar ${\beta }_t\in [0,1]$ is the noise variance at step $t$, chosen by us before training starts; the collection ${\beta }_1,\dots ,{\beta }_T$ is called the noise schedule. Define the retention factor ${\alpha }_t=1-{\beta }_t$: the fraction of the old signal that survives each step. Because ${\alpha }_t$ appears at every step, it helps to give its running product a name: the cumulative product

$${\bar{\alpha }}_t=\prod _{s=1}^t{\alpha }_s={\alpha }_1{\alpha }_2\cdots {\alpha }_t$$

Since every factor satisfies ${\alpha }_s<1$, the product ${\bar{\alpha }}_t$ shrinks toward zero as $t$ grows.

This looks like a recursion that walks ${\mathbf{x}}_0\to {\mathbf{x}}_1\to {\mathbf{x}}_2\to \dots$, but because Gaussian noise added to Gaussian noise stays Gaussian, the whole chain collapses into one expression in terms of ${\mathbf{x}}_0$. Here is why, worked out explicitly. Take the first two steps and substitute the first into the second:

$$\begin{array}{rlr}{\mathbf{x}}_2& =\sqrt{{\alpha }_2}{\mathbf{x}}_1+\sqrt{{\beta }_2}{\mathbf{ϵ}}_2& \\ & =\sqrt{{\alpha }_2}(\sqrt{{\alpha }_1}{\mathbf{x}}_0+\sqrt{{\beta }_1}{\mathbf{ϵ}}_1)+\sqrt{{\beta }_2}{\mathbf{ϵ}}_2& \text{(substitute }{\mathbf{x}}_1\text{)}\\ & =\sqrt{{\alpha }_1{\alpha }_2}{\mathbf{x}}_0+\underset{\text{two independent zero-mean Gaussians}}{\underbrace{\sqrt{{\alpha }_2{\beta }_1}{\mathbf{ϵ}}_1+\sqrt{{\beta }_2}{\mathbf{ϵ}}_2}}& \end{array}$$

A sum of two independent zero-mean Gaussians is itself a zero-mean Gaussian whose variance is the sum of the variances. The combined variance here is ${\alpha }_2{\beta }_1+{\beta }_2$. Now check that this is exactly what is missing from full retention: $1-{\alpha }_1{\alpha }_2=({\alpha }_1{\alpha }_2+{\beta }_1{\alpha }_2+{\beta }_2)-{\alpha }_1{\alpha }_2={\alpha }_2{\beta }_1+{\beta }_2$. So the two noise terms fuse into a single equivalent draw $\mathbf{ϵ}\sim \mathcal{N}(0,\mathbf{I})$:

$${\mathbf{x}}_t=\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0+\sqrt{1-{\bar{\alpha }}_t}\mathbf{ϵ},\mathbf{ϵ}\sim \mathcal{N}(0,\mathbf{I})$$

Reading it aloud: the new sample equals the square root of the cumulative retention coefficient times the original sample, plus the square root of one minus that coefficient times a fresh noise draw. (Notation note: some references write the same equation as ${\mathbf{z}}_t=\sqrt{{\bar{\alpha }}_t}\mathbf{x}+\sqrt{1-{\bar{\alpha }}_t}\mathbf{ϵ}$ using latent-variable names; the content is identical.)

14.1.2 The Closed-Form Distribution

Because the collapsed expression above is a linear function of ${\mathbf{x}}_0$ plus a Gaussian, the conditional distribution of ${\mathbf{x}}_t$ given the original data point is itself Gaussian:

$$q({\mathbf{x}}_t\mid {\mathbf{x}}_0)=\mathcal{N}\left({\mathbf{x}}_t;\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0,(1-{\bar{\alpha }}_t)\mathbf{I}\right)$$

The mean sits at $\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0$ and the covariance comes from $1-{\bar{\alpha }}_t$. The matrix $\mathbf{I}\in {\mathbb{R}}^{d\times d}$ is the identity covariance matrix, with unit diagonal entries, so it looks like a long row of ones matching the dimension $d$ of your image — each pixel gets its own independent share of noise. This distribution is sometimes called the diffusion kernel: given a fixed starting point ${\mathbf{x}}_0$, it tells you the cloud of possible noisy versions at step $t$.

Worked example. Suppose the schedule is constant with ${\beta }_1=0.1$ and ${\beta }_2=0.2$, and the data is a single pixel $x_0=1.0$. Then ${\alpha }_1=0.9$, ${\alpha }_2=1-0.2=0.8$, and the cumulative product is ${\bar{\alpha }}_2=0.9\times 0.8=0.72$. Jumping straight to step 2 gives mean $\sqrt{0.72}\times 1.0\approx 0.849$ and variance $1-0.72=0.28$.

Walking instead: $\mathbb{E}[{\mathbf{x}}_1]=\sqrt{0.9}\times 1.0\approx 0.949$, then $\mathbb{E}[{\mathbf{x}}_2]=\sqrt{0.8}\times 0.949\approx 0.849$. Same mean, and the variance accumulated along the way is ${\alpha }_2{\beta }_1+{\beta }_2=0.8\times 0.1+0.2=0.28$. Both routes agree — the jump is exact, not an approximation, because the final answer is mean $\approx 0.849$, variance $0.28$ either way. Sense-check: the variance $0.28>0$ and less than 1, and the mean shrank toward zero, as expected when the signal keeps being attenuated.

14.1.3 Limit Behavior and the Shortcut

Push $t$ very large and the product ${\bar{\alpha }}_t$ decays to zero, because you keep multiplying numbers below one. Then the mean term vanishes and:

$$q({\mathbf{x}}_T\mid {\mathbf{x}}_0)⟶\mathcal{N}(0,\mathbf{I})$$

The signal is destroyed completely; what survives is pure noise with zero mean and unit covariance. Notice the boundary behavior: at $t=0$ there is no noise at all, and at $t=T$ there is no signal at all. Everything between is a smooth blend whose mixing ratio is set by ${\bar{\alpha }}_t$ alone.

The practical gift is the shortcut: this closed form lets you jump straight to any timestep. You never need to walk through ${\mathbf{x}}_1,{\mathbf{x}}_2,\dots$ one at a time; you can leap from ${\mathbf{x}}_0$ to any ${\mathbf{x}}_i$ or ${\mathbf{x}}_j$ in one shot. During training this matters enormously — for each training image we pick a random timestep $t$, synthesize ${\mathbf{x}}_t$ in a single line of algebra, and ask the network to denoise it. Without the shortcut, every training example would cost up to $T$ sequential noising operations.

Real-world connection: this forward process doubles as an infinite data augmentation engine. Every training image can be paired with unlimited randomly-noised versions at random timesteps — the same photo becomes thousands of distinct regression targets. That is one reason diffusion training is famously stable compared with GAN training: the network always sees a supervised target (the exact noise added), much like medical-imaging pipelines that train segmentation networks on artificially degraded scans so they learn to restore real ones.

14.2 The Reverse Denoising Process

14.2.1 What We Want to Learn

The reverse process starts at the far end. Draw ${\mathbf{x}}_T$ from the standard Gaussian $\mathcal{N}(0,\mathbf{I})$ — remember its dimension matches the image size, or the size of a text embedding, whatever you are generating. Then learn to remove noise step by step. After ample noise removal you should land on an image that looks like it came from the training distribution.

So the object of interest is the reverse transition $q({\mathbf{x}}_{t-1}\mid {\mathbf{x}}_t)$: given a noisy sample, which slightly cleaner sample produced it?

14.2.2 Intractable Alone, Tractable with the Data Point

Intuition + analogy. Imagine restoring an old, grainy photograph pixel by pixel. If the original photo sits beside you, every restoration step is easy: you know exactly what the clean version should look like, so you can check how much grain to peel off. Take the original away and the same job becomes guesswork — thousands of different originals could plausibly explain the same faded print. The original photo is ${\mathbf{x}}_0$. With it in hand, denoising is a well-defined problem; without it, it is ambiguous.

Why exactly does removing ${\mathbf{x}}_0$ make things hard? Learning the reverse transition alone stays intractable — no closed form exists without knowing where the data lives. Apply Bayes' rule to the quantity we want:

$$q({\mathbf{x}}_{t-1}\mid {\mathbf{x}}_t)=\frac{q({\mathbf{x}}_t\mid {\mathbf{x}}_{t-1})q({\mathbf{x}}_{t-1})}{q({\mathbf{x}}_t)}$$

Every piece on the right involves either the data distribution $q({\mathbf{x}}_{t-1})$ or the marginal $q({\mathbf{x}}_t)$, each computed by averaging over all possible clean images — and we do not know the data distribution; estimating it is the whole problem we started with. So this expression cannot be computed. Knowing ${\mathbf{x}}_0$ fixes it.

Now condition on the data point instead. By Bayes' rule again,

$$q({\mathbf{x}}_{t-1}\mid {\mathbf{x}}_t,{\mathbf{x}}_0)=\frac{q({\mathbf{x}}_t\mid {\mathbf{x}}_{t-1})q({\mathbf{x}}_{t-1}\mid {\mathbf{x}}_0)}{q({\mathbf{x}}_t\mid {\mathbf{x}}_0)}$$

and suddenly every factor is known: $q({\mathbf{x}}_t\mid {\mathbf{x}}_{t-1})$ is the fixed forward step, and both $q({\mathbf{x}}_{t-1}\mid {\mathbf{x}}_0)$ and $q({\mathbf{x}}_t\mid {\mathbf{x}}_0)$ are closed-form Gaussians from the previous section. Two Gaussians combined this way produce another Gaussian — so the answer has an exact formula:

$$q({\mathbf{x}}_{t-1}\mid {\mathbf{x}}_t,{\mathbf{x}}_0)\text{is tractable, while}q({\mathbf{x}}_{t-1}\mid {\mathbf{x}}_t)\text{is not}$$

A concrete taste of the ambiguity: suppose one pixel starts from $x_0=+2$ or from $x_0^{\prime }=-2$, with per-step retention $\sqrt{{\bar{\alpha }}_1}\approx 0.95$. Both starting points produce noisy values centered near $+1.9$ and $-1.9$ — but with Gaussian spread around them. An observation $x_t=0.3$ is entirely consistent with either origin; the conditional distribution $q(x_{t-1}\mid x_t=0.3)$ therefore mixes two very different humps (one near positive values, one near negative), which is why it is not a clean single Gaussian. Fix $x_0=+2$, however, and only the first hump survives — a tidy Gaussian remains.

14.2.3 Parameterizing the Reverse Chain

Set the starting point of the chain as the fixed prior $p({\mathbf{x}}_T)=\mathcal{N}(0,\mathbf{I})$ — chosen deliberately to match where the forward process ends, so the two chains meet cleanly in the middle. Then learn a parameterized reverse transition for every step and multiply the chain:

$$p_{\theta }({\mathbf{x}}_{0:T})=p({\mathbf{x}}_T)\prod _{t=1}^T{p}_{\theta }({\mathbf{x}}_{t-1}\mid {\mathbf{x}}_t)$$

Reading the notation: $\mathbf{\theta }$ (theta) is the collection of all network weights being learned, and ${\mathbf{x}}_{0:T}$ abbreviates the whole sequence ${\mathbf{x}}_0,{\mathbf{x}}_1,\dots ,{\mathbf{x}}_T$. Each reverse step is written as a Gaussian whose mean vector ${\mathbf{\mu }}_{\theta }\in {\mathbb{R}}^d$ and covariance ${\mathbf{\Sigma }}_{\theta }\in {\mathbb{R}}^{d\times d}$ come from a network:

$$p_{\theta }({\mathbf{x}}_{t-1}\mid {\mathbf{x}}_t)=\mathcal{N}\left({\mathbf{x}}_{t-1};{\mathbf{\mu }}_{\theta }({\mathbf{x}}_t,t),{\mathbf{\Sigma }}_{\theta }({\mathbf{x}}_t,t)\right)$$

Both quantities are time-varying: they change with $t$ and depend on the current noisy sample ${\mathbf{x}}_t$. Why is a Gaussian a fair approximation for the true reverse step? Because the schedule keeps each ${\beta }_t$ small, one denoising move covers only a short distance, and over such short distances the messy multi-modal reverse distribution is close to normal — the same reason a short stretch of any smooth curve looks like a straight line. A deep network takes ${\mathbf{x}}_t$ and $t$ and estimates the mean (and variance) so that the likelihood of the data is maximized.

Unrolling the product makes sampling concrete — the procedure called ancestral sampling: draw ${\mathbf{x}}_T$ from the prior, sample ${\mathbf{x}}_{T-1}$ from $p_{\theta }({\mathbf{x}}_{T-1}\mid {\mathbf{x}}_T)$, then ${\mathbf{x}}_{T-2}$, continuing down to the final transition $p_{\theta }({\mathbf{x}}_0\mid {\mathbf{x}}_1)$. One full generation costs $T$ sequential network evaluations.

Real-world connection: ancestral sampling through learned Gaussian steps is the engine behind modern image synthesizers — every image produced by tools like Stable Diffusion is the endpoint of exactly such a chain, evaluated sequentially on a GPU cluster. The same reverse-chain machinery also drives medical image reconstruction, where diffusion priors restore MRI scans from partial measurements by iteratively removing corruption guided by a learned model.

14.3 Training Objective: A Hierarchical VAE View

14.3.1 From Likelihood to KL Divergences

Training maximizes the log likelihood of the data. Taking $\ln$ turns the long product of transitions into a sum, which is far easier to optimize. Work the variational bound and everything boils down to minimizing Kullback-Leibler (KL) divergence — a measure of how badly two distributions disagree; it is zero when they match and grows the further apart they sit. Concretely: for each step, the learned reverse transition $p_{\theta }({\mathbf{x}}_{t-1}\mid {\mathbf{x}}_t)$ should match the forward posterior $q({\mathbf{x}}_{t-1}\mid {\mathbf{x}}_t,{\mathbf{x}}_0)$.

The derivation mirrors the VAE bound closely. Recall the VAE. One encoder produced a mean and variance, and a KL term pulled the latent distribution toward a standard Gaussian. You sampled from that Gaussian, the decoder reconstructed the data, and the objective maximized the likelihood of the reconstruction. Diffusion follows the same recipe, applied at every step — so think of it as a hierarchical VAE. Instead of one encoder you stack several. Encoder one learns a mean and variance from the data, encoder two conditions on that output, and so on. Decoding then unwinds hierarchically back to the reconstruction.

One structural twist matters: in a plain VAE both halves train together, but here the encoder is predetermined — so all learned parameters live in the decoder side, which must do the work of tightening the bound.

14.3.2 The Three ELBO Terms

Expanding the bound splits it into three terms. The spoken structure: the reconstruction piece measures going from ${\mathbf{x}}_1$ to ${\mathbf{x}}_0$, the prior matching piece compares the fully noised sample against the standard Gaussian, and the remaining pieces form the denoising matching term:

$$\log p_{\theta }({\mathbf{x}}_0)\ge \underset{\text{reconstruction}}{\underbrace{{\mathbb{E}}_q\left[\log p_{\theta }({\mathbf{x}}_0\mid {\mathbf{x}}_1)\right]}}-\underset{\text{prior matching}}{\underbrace{D_{\mathrm{KL}}\left(q({\mathbf{x}}_T\mid {\mathbf{x}}_0)\Vert p({\mathbf{x}}_T)\right)}}-\sum _{t=2}^T\underset{\text{denoising matching}}{\underbrace{D_{\mathrm{KL}}\left(q({\mathbf{x}}_{t-1}\mid {\mathbf{x}}_t,{\mathbf{x}}_0)\Vert p_{\theta }({\mathbf{x}}_{t-1}\mid {\mathbf{x}}_t)\right)}}$$

(Index note: the sum runs over $t=2,\dots ,T$ — the intermediate steps only. The first reverse move into the data, $t=1$, is separated out as the reconstruction term, and the last forward step is separated out as the prior matching term.) Walk through the pieces:

• Reconstruction term. Apply the learned reverse step one last time, $p_{\theta }({\mathbf{x}}_0\mid {\mathbf{x}}_1)$, and reward it for rebuilding the data. Higher is better. Like a VAE decoder's final output layer, it turns the almost-clean sample back into pixels.
• Prior matching term. $q({\mathbf{x}}_T\mid {\mathbf{x}}_0)$ is known in closed form, and after enough noise additions the signal is lost completely — only noise remains — so this term simply checks that the end of the forward chain agrees with the standard Gaussian $p({\mathbf{x}}_T)=\mathcal{N}(0,\mathbf{I})$. No learned parameters here; with a long enough chain it is essentially zero for free, which is why the design insists on enough steps.
• Denoising matching term. The sum of KL divergences across intermediate steps. Minimizing it forces each learned denoising transition to imitate the tractable forward posterior. This term is trainable, and it is tractable precisely because $q({\mathbf{x}}_{t-1}\mid {\mathbf{x}}_t,{\mathbf{x}}_0)$ has a closed form when ${\mathbf{x}}_0$ is available — and during training, ${\mathbf{x}}_0$ always is available, because we hold the training image in our hands.

Minimizing this average divergence teaches the denoising function that takes ${\mathbf{x}}_t$ and returns ${\mathbf{x}}_{t-1}$. That is the key training idea of diffusion. Learning a good $p_{\theta }({\mathbf{x}}_{t-1}\mid {\mathbf{x}}_t)$ maximizes the likelihood of reconstructed training data, so diffusion sits firmly in the likelihood-maximization family, like the DAE and VAE ideas before it.

14.3.3 The Tractable Posterior and Noise Prediction

Everything now hinges on writing down $q({\mathbf{x}}_{t-1}\mid {\mathbf{x}}_t,{\mathbf{x}}_0)$ explicitly. Start from Bayes' rule conditioned on the data:

$$q({\mathbf{x}}_{t-1}\mid {\mathbf{x}}_t,{\mathbf{x}}_0)=\frac{q({\mathbf{x}}_t\mid {\mathbf{x}}_{t-1})q({\mathbf{x}}_{t-1}\mid {\mathbf{x}}_0)}{q({\mathbf{x}}_t\mid {\mathbf{x}}_0)}$$

All three factors are closed-form Gaussians from Section 14.1. Multiplying two Gaussian densities in the same variable produces another Gaussian (complete-the-square), so the posterior is itself a Gaussian with a computable mean and variance:

$$q({\mathbf{x}}_{t-1}\mid {\mathbf{x}}_t,{\mathbf{x}}_0)=\mathcal{N}\left({\mathbf{x}}_{t-1};{\tilde{\mathbf{\mu }}}_t({\mathbf{x}}_t,{\mathbf{x}}_0),{\tilde{\beta }}_t\mathbf{I}\right)$$

Its mean blends the original data point and the current noisy sample:

$${\tilde{\mathbf{\mu }}}_t({\mathbf{x}}_t,{\mathbf{x}}_0)=\frac{\sqrt{{\bar{\alpha }}_{t-1}}{\beta }_t}{1-{\bar{\alpha }}_t}{\mathbf{x}}_0+\frac{\sqrt{{\alpha }_t}(1-{\bar{\alpha }}_{t-1})}{1-{\bar{\alpha }}_t}{\mathbf{x}}_t\text{with}{\tilde{\beta }}_t=\frac{{\beta }_t(1-{\bar{\alpha }}_{t-1})}{1-{\bar{\alpha }}_t}$$

Here ${\beta }_t$ is the per-step noise variance and ${\alpha }_t=1-{\beta }_t$, with the cumulative product ${\bar{\alpha }}_t={\prod }_{s=1}^t{\alpha }_s$ defined above. (Notation note: some references write the second numerator as $\sqrt{1-{\beta }_t}(1-{\bar{\alpha }}_{t-1})$; since $\sqrt{1-{\beta }_t}=\sqrt{{\alpha }_t}$, the forms are identical.) Two sanity checks before moving on. Boundary check: set $t=1$ with the convention ${\bar{\alpha }}_0=1$ — the first coefficient becomes ${\beta }_1/(1-{\bar{\alpha }}_1)={\beta }_1/{\beta }_1=1$ and the second becomes zero, so the posterior mean collapses to ${\mathbf{x}}_0$ itself, exactly right for the step adjacent to clean data. Weight check: the two coefficients stay below one, with the leftover mass absorbed by the posterior variance ${\tilde{\beta }}_t$.

Rather than predicting the mean directly, the usual move is a noise prediction network. Rewrite the mean so the only unknown is the noise that was added. First solve the forward equation for ${\mathbf{x}}_0$: from ${\mathbf{x}}_t=\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0+\sqrt{1-{\bar{\alpha }}_t}\mathbf{ϵ}$,

$${\mathbf{x}}_0=\frac{1}{\sqrt{{\bar{\alpha }}_t}}({\mathbf{x}}_t-\sqrt{1-{\bar{\alpha }}_t}\mathbf{ϵ})$$

Substitute this into the blended mean above and simplify (using $\sqrt{{\bar{\alpha }}_{t-1}}/\sqrt{{\bar{\alpha }}_t}=1/\sqrt{{\alpha }_t}$ and ${\alpha }_t+{\beta }_t=1$):

$$\begin{array}{rlr}{\tilde{\mathbf{\mu }}}_t& =\frac{\sqrt{{\bar{\alpha }}_{t-1}}{\beta }_t}{1-{\bar{\alpha }}_t}\cdot \frac{{\mathbf{x}}_t-\sqrt{1-{\bar{\alpha }}_t}\mathbf{ϵ}}{\sqrt{{\bar{\alpha }}_t}}+\frac{\sqrt{{\alpha }_t}(1-{\bar{\alpha }}_{t-1})}{1-{\bar{\alpha }}_t}{\mathbf{x}}_t& \text{(substitute }{\mathbf{x}}_0\text{)}\\ & =\frac{{\beta }_t/\sqrt{{\alpha }_t}}{1-{\bar{\alpha }}_t}{\mathbf{x}}_t+\frac{\sqrt{{\alpha }_t}(1-{\bar{\alpha }}_{t-1})}{1-{\bar{\alpha }}_t}{\mathbf{x}}_t-\frac{{\beta }_t}{\sqrt{{\alpha }_t}\sqrt{1-{\bar{\alpha }}_t}}\mathbf{ϵ}& \text{(collect }{\mathbf{x}}_t\text{ terms)}\\ & =\frac{{\beta }_t+{\alpha }_t-{\bar{\alpha }}_t}{\sqrt{{\alpha }_t}(1-{\bar{\alpha }}_t)}{\mathbf{x}}_t-\frac{{\beta }_t}{\sqrt{{\alpha }_t}\sqrt{1-{\bar{\alpha }}_t}}\mathbf{ϵ}& \text{(common denominator; }{\alpha }_t{\bar{\alpha }}_{t-1}={\bar{\alpha }}_t\text{)}\\ & =\frac{1}{\sqrt{{\alpha }_t}}{\mathbf{x}}_t-\frac{{\beta }_t}{\sqrt{{\alpha }_t}\sqrt{1-{\bar{\alpha }}_t}}\mathbf{ϵ}& \text{(since }{\beta }_t+{\alpha }_t=1\text{)}\end{array}$$

Every quantity except $\mathbf{ϵ}$ is known at sampling time, so replace the true noise with a network's guess:

$${\mathbf{\mu }}_{\theta }({\mathbf{x}}_t,t)=\frac{1}{\sqrt{{\alpha }_t}}\left({\mathbf{x}}_t-\frac{{\beta }_t}{\sqrt{1-{\bar{\alpha }}_t}}{\mathbf{ϵ}}_{\theta }({\mathbf{x}}_t,t)\right)$$

The network ${\mathbf{ϵ}}_{\theta }$, with parameters $\theta$, predicts the noise $\mathbf{ϵ}$ at every timestep. You find the best $\theta$ by maximizing the likelihood of the training data through the bound above. Why prefer this parameterization? Because the target — the exact noise that was mixed in — is known perfectly at training time, which converts a distribution-matching problem into plain regression.

14.3.4 The Simple Loss

Because both sides of each KL are Gaussians, the KL between them has a closed form: it reduces to (a constant times) the squared distance between the two means plus constants that do not involve $\theta$. Substituting the reparameterized mean, those surviving squared distances turn out to be weighted copies of one comparison — actual noise versus predicted noise — and dropping the timestep-dependent weights (an empirical choice that works better in practice) leaves the celebrated simple objective:

$$L_t=\Vert \mathbf{ϵ}-{\mathbf{ϵ}}_{\theta }({\mathbf{x}}_t,t){\Vert }^2$$

Here $\Vert \cdot {\Vert }^2$ sums squared differences over every pixel channel — for an $H\times W$ color image it is a sum over $3HW$ numbers. That is the whole training signal: draw a noisy sample, ask the network what noise was mixed in, and penalize the squared gap.

Real-world connection: this "predict the corruption" recipe is used far beyond art generation — seismic imaging teams train networks on synthetic noise-corrupted signals to denoise field recordings, and speech-enhancement systems regress the additive noise spectrum of phone calls. Diffusion models industrialized the idea: the corruption is a controlled Gaussian whose exact value is always known, giving unlimited perfectly-labeled regression targets.

14.4 The U-Net Noise Prediction Network

14.4.1 Quick CNN Recap

Which network plays the role of ${\mathbf{ϵ}}_{\theta }$? The answer is the U-Net architecture, and since not everyone had met it, the session backed up briefly.

That is image-level classification. Diffusion needs something else: at every denoising step we must transform an entire image-shaped tensor into another image-shaped tensor — a per-pixel job. A classification head that squashes everything into one label throws away exactly the spatial layout diffusion must preserve.

14.4.2 How U-Net Relates to the Convolutional Autoencoder

A U-Net is built from convolution layers, arranged like a deep convolutional autoencoder. Convolution filters press the input down from a high-dimensional image into a small latent representation. Transposed convolutions (learned upsampling layers that stretch feature maps back by inserting zeros and applying filters) then expand it back to the original size. You can fairly view a U-Net as a special case of that autoencoder family. The goals differ, though:

When to pick which: reach for an autoencoder when you want a compact representation of an image; reach for a U-Net when you want an image-sized answer computed from an image-sized input.

Originally the U-Net was built for biomedical image processing, where the task was segmentation: split an image into groups of pixels. Picture a photo where a cat sits on a sofa. A trained U-Net assigns every cat pixel a cat label, the sofa region a sofa label, and everything else a background label — pixel-level answers rather than one caption for the whole picture. The same "answer at every pixel" property is precisely what noise prediction needs: the noise added to pixel $(i,j)$ is a number attached to $(i,j)$, so the predictor must speak in pixels too.

14.4.3 Down Path, Up Path, and Skip Connections

Shape-wise, an input image is $H\times W\times C$ — height $H$, width $W$, and $C=3$ channels for color. The down path applies max pooling or strided convolutions, shrinking spatial size while building a compact representation at the bottleneck. As space shrinks, the number of channels typically grows, trading resolution for semantics. The up path then uses transposed convolutions to grow the map back to full pixel-level resolution while channels shrink again.

The signature move is the skip connection: feature maps from the down path are concatenated with the corresponding up-path layers, tying the paired layers together. Each level thereby carries coarse semantic information and fine spatial detail at once. Why is that pairing essential? The bottleneck compresses so hard that fine details (exact edge positions, individual whiskers) cannot survive the trip down and back on their own; the skips give them a shortcut around the squeeze. That is what makes pixel-sharp outputs possible. Attention mechanisms can also be bolted onto the U-Net for extra power, but the core stays fixed: a bundle of ordinary convolution layers plus matching transposed convolutions that transform every pixel.

Inside diffusion training, this network estimates ${\mathbf{ϵ}}_{\theta }({\mathbf{x}}_t,t)$. Feed it ${\mathbf{x}}_t$; subtract its scaled noise estimate to get ${\mathbf{x}}_{t-1}$; feed that in again with the next time value; repeat until ${\mathbf{x}}_0$.

14.4.4 Time Conditioning Through Sinusoidal Embeddings

One requirement is easy to miss: the network must know which denoising step it is on. Early steps face a nearly pure-noise canvas and should make broad structural guesses; late steps face nearly finished images and should fix fine grain. The same input values demand different answers at different times, so time must enter the computation. The fix is to embed the scalar time $t$ — a fully connected layer turns, say, $t=2000$ into a vector. That embedding goes in as a condition to every block of the U-Net. Next round, the embedding for $t=1999$ enters, and so on down to the last step. One shared network serves all timesteps — training and storing thousands of separate networks would be hopeless.

Typically the time representation uses sinusoidal positional embeddings, the same device used to encode position in attention models, as covered in the deep neural networks material. For an embedding vector of width $d_{\text{emb}}$, the values are fixed sines and cosines at geometrically spaced frequencies:

$$\text{PE}(t,2i)=\sin \left(\frac{t}{10000^{2i/d_{\text{emb}}}}\right),\text{PE}(t,2i+1)=\cos \left(\frac{t}{10000^{2i/d_{\text{emb}}}}\right),i=0,1,\dots$$

Fast-varying components separate neighboring timesteps; slow-varying ones keep distant timesteps distinct. The embedding rides along with the image through the whole network — usually added to the feature maps at every stage, so every layer can consult the clock.

Real-world connection: beyond its original home in cell microscopy segmentation (where it delineated cell boundaries in microscope imagery), the U-Net is now the computational workhorse inside every major diffusion-based image generator — and variants run in medical imaging pipelines for organ contouring in radiotherapy planning, where pixel-exact outputs are a clinical requirement, not a luxury.

14.5 Measuring Generation Quality: FID and IS

14.5.1 Two Scores to Watch

Trained with the squared noise loss, diffusion models reach very low FID scores and reasonably high inception scores. Both metrics appeared already in the GAN material and in your assignments, and they answer different questions.

• FID (Fréchet Inception Distance): smaller is better. A small FID signals that the output spread is genuinely diverse and is not collapsing onto a few modes — it is not the result of mode collapse.
• IS (Inception Score): bigger is better. A high IS says the generated images are of high quality and sit close to the images the model trained on.

Keep the pairing in mind when you tune generative systems: one score polices diversity, the other polices fidelity, and a strong model needs both.

How do they work, mechanically? Both lean on a fixed pretrained ImageNet classifier called Inception.

• Inception Score. Each generated image is classified. Two things should hold at once: each individual image should produce a peaked class distribution $\mathrm{Pr}(y\mid {\mathbf{x}}_i^{\ast })$ — the picture looks confidently like one thing — while averaged over the whole generated set, the class distribution $\mathrm{Pr}(y)$ should be flat, meaning every class appears about equally often. The score combines both demands into one number:

$$\mathrm{IS}=\exp \left(\frac{1}{I}\sum _{i=1}^I{D}_{\mathrm{KL}}\left(\mathrm{Pr}(y\mid {\mathbf{x}}_i^{\ast })\Vert \mathrm{Pr}(y)\right)\right)$$

where $I$ is the number of generated images and $\mathrm{Pr}(y)=\frac{1}{I}{\sum }_{i=1}^I\mathrm{Pr}(y\mid {\mathbf{x}}_i^{\ast })$. The KL divergence inside is large exactly when per-image predictions are sharp but the overall mix is spread out; exponentiating turns it into a friendlier multiplicative scale.

• Fréchet Inception Distance. Instead of looking at images one at a time, FID asks whether the cloud of generated images resembles the cloud of real images. Both clouds are passed through Inception, described by their mean vectors and covariance matrices in its feature space, and the distance between the two Gaussians is computed with the closed-form Fréchet distance. If the two clouds overlap well, FID is small; if the generator covers only part of the real cloud (mode collapse) or produces blurry off-manifold points, FID grows.

One-sentence rule for choosing: use IS for a quick read on sample recognizability, trust FID when diversity versus the real distribution matters — and report both when you can.

14.5.2 What the Loss Buys You

The ELBO-driven loss pushes the network to reproduce image details faithfully. That is the basic loop of reverse diffusion training: sample something from the standard Gaussian, peel off noise step by step, and drive the process so the likelihood of the denoised, generated data is maximized. Because every training target is a supervised regression problem, nothing in this loop fights back the way an adversarial discriminator does — which is why diffusion models reached state-of-the-art FID numbers without the instability GANs are famous for.

A fair caveat from the metric literature: both scores judge through Inception's eyes. Whatever information the classifier's features discard cannot influence either number, so a model could score well while failing on details Inception never measures. That is why qualitative eyeballing still accompanies every quantitative claim.

Real-world connection: any team deploying generative models in production tracks FID across releases to catch silent regressions — stock-photo generators, game-asset pipelines, and data-augmentation engines all gate updates on FID not worsening, because a mode-collapsed model quietly shrinks the creative range of everything built on top of it.

14.6 Alternate Parameterization: DDIM

14.6.1 Predicting Data Instead of Noise

Everything so far predicted noise. An alternate parameterization predicts the original data instead. The training process stays the same — same forward noising, same network, same squared loss on noise — but generation can take a shortcut. Once the network outputs ${\mathbf{ϵ}}_{\theta }({\mathbf{x}}_t,t)$, rearrange the forward equation to recover an estimate of the clean image. Start from the closed form ${\mathbf{x}}_t=\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0+\sqrt{1-{\bar{\alpha }}_t}\mathbf{ϵ}$ and solve for ${\mathbf{x}}_0$: subtract the noise term from both sides, then divide by its coefficient:

$${\hat{\mathbf{x}}}_{\theta }({\mathbf{x}}_t,t)=\frac{1}{\sqrt{{\bar{\alpha }}_t}}\left({\mathbf{x}}_t-\sqrt{1-{\bar{\alpha }}_t}{\mathbf{ϵ}}_{\theta }({\mathbf{x}}_t,t)\right)$$

Because you hold both ${\mathbf{x}}_t$ and the predicted noise, you can compute this estimate at any step — and that unlocks jumping. Instead of marching $1000,999,998,\dots$ you can hop ${\mathbf{x}}_{1000}\to {\mathbf{x}}_{500}\to {\mathbf{x}}_{100}\to {\mathbf{x}}_{50}\to {\mathbf{x}}_{20}\to {\mathbf{x}}_0$. Fewer denoising steps means faster generation. This implicit, stepped variant is the DDIM — Denoising Diffusion Implicit Model. Training is identical for the probabilistic and implicit variants; at generation time DDIM uses data prediction, which also makes the trajectory more deterministic — run the same starting noise twice and you land on essentially the same image, unlike the fully stochastic chain.

14.6.2 Training Algorithm

This is a loop you could code in an afternoon — which is a large part of why diffusion took over.

Purpose: teach one shared U-Net to name the exact noise mixed into any image at any corruption level. Inputs: training images ${\mathbf{x}}_0$; schedule constants ${\beta }_t$ (so ${\alpha }_t,{\bar{\alpha }}_t$ follow); batch size. Outputs: parameters $\theta$ of the noise predictor ${\mathbf{ϵ}}_{\theta }({\mathbf{x}}_t,t)$.

The procedure, step by step:

1. Draw a training sample ${\mathbf{x}}_0$ from the dataset.
2. Pick a timestep $t$ uniformly at random from $1,\dots ,T$.
3. Draw Gaussian noise $\mathbf{ϵ}\sim \mathcal{N}(0,\mathbf{I})$. Treat it as a vector with the same dimension as the image — not a scalar.
4. Form the noisy sample with the closed form, ${\mathbf{x}}_t=\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0+\sqrt{1-{\bar{\alpha }}_t}\mathbf{ϵ}$, and ask the U-Net for its guess ${\mathbf{ϵ}}_{\theta }({\mathbf{x}}_t,t)$.
5. Minimize $\Vert \mathbf{ϵ}-{\mathbf{ϵ}}_{\theta }({\mathbf{x}}_t,t){\Vert }^2$ over all U-Net parameters $\theta$.

It really is a regression problem: estimate the exact noise that got mixed in. Cost note: each iteration costs one forward pass, one backward pass, and one closed-form noising — no sequential unrolling, which is why training is cheap relative to sampling.

14.6.3 Sampling Algorithm

Generation runs the loop in reverse:

1. Sample ${\mathbf{x}}_T\sim \mathcal{N}(0,\mathbf{I})$.
2. For $t=T,T-1,\dots ,1$: prepare a noise vector $\mathbf{z}$ with image dimension — sample it fresh when $t>1$; set $\mathbf{z}=0$ at the final step $t=1$.
3. Combine the U-Net output ${\mathbf{ϵ}}_{\theta }({\mathbf{x}}_t,t)$ with $\mathbf{z}$ to compute ${\mathbf{x}}_{t-1}$, then repeat with the new sample.
4. Stop at ${\mathbf{x}}_0$.

Why zero the last $\mathbf{z}$? The final transition should commit to a single clean image rather than shake the result one more time — after all the refinement, extra randomness would only blur the finish. Notice the rhythm: the U-Net consumes ${\mathbf{x}}_t$, emits a noise estimate, that estimate plus $\mathbf{z}$ produces ${\mathbf{x}}_{t-1}$, and the same trained U-Net runs again on the cleaner input, all the way down.

With the DDIM variant, the same skeleton runs over a sub-sequence of timesteps (the hop list above), each update built from the recovered ${\hat{\mathbf{x}}}_{\theta }$; because updates are deterministic, large hops stay accurate enough to keep quality high while cutting the number of U-Net calls by an order of magnitude or more.

14.6.4 The Speed Trade-Off

Honesty demands acknowledging the weakness: image generation with diffusion is slow. Anyone who has asked a chatbot to edit an uploaded image has felt the wait, because such tools lean on diffusion-like incremental refinement. The vanilla formulation is nearly a Markov chain — each removal depends on the immediately preceding step, so $t=1000$ means a thousand sequential passes through a heavy network, and sequential passes cannot be parallelized across time. DDIM softens this by skipping steps, but iteration remains inherent. One-shot generators do not pay this cost — more on that in a moment.

Real-world connection: every consumer tool that edits an uploaded photo through a chatbot feels this trade-off directly — the wait you experience is a stack of sequential denoising calls on a server GPU. Engineering teams ship DDIM-style samplers precisely because cutting 1000 calls to 20–50 turns seconds-long waits into something interactive.

14.7 Comparing the Generative Families

14.7.1 VAE Pipeline

Given data, the encoder learns a latent distribution; minimizing KL divergence keeps that distribution close to a standard Gaussian. Sampling then draws

$$\mathbf{z}=\mathbf{\mu }+\mathbf{\sigma }\odot \mathbf{ϵ},\mathbf{ϵ}\sim \mathcal{N}(0,\mathbf{I})$$

— the learned mean $\mathbf{\mu }\in {\mathbb{R}}^k$ plus the learned standard deviation $\mathbf{\sigma }\in {\mathbb{R}}^k$ times a noise draw $\mathbf{ϵ}\in {\mathbb{R}}^k$; the symbol $\odot$ is elementwise multiplication and the resulting $\mathbf{z}\in {\mathbb{R}}^k$ lives in a compact latent space you chose at design time. The decoder maps $\mathbf{z}$ to a reconstruction $\bar{\mathbf{x}}$. Training maximizes the likelihood of the reconstruction while the KL term holds the latent space well-behaved — every region decodes into something plausible, so any fresh draw is usable.

14.7.2 Diffusion Pipeline

Start from data ${\mathbf{x}}_0$, add noise repeatedly until reaching total noise — a Gaussian. Then learn the reverse process: a noise prediction network trained by likelihood-based fitting, so that whatever it finally spits out has maximal data probability. Sampling begins at a draw from the normal distribution, and because each run draws a different sample, applying noise reduction to different draws yields different outcomes. The defining structural fact: that starting draw has the same dimension as the image itself — nothing is compressed anywhere along the way.

14.7.3 GAN Pipeline

Feed a small random code into the generator; train a discriminator to tell generated samples from training data. Generation happens in one shot: the code passes through a stack of convolution layers and the full image emerges at once. No likelihood, no iterative refinement — just a straight conveyor belt from noise vector to picture.

14.7.4 The Latent-Dimension Question

Then came a question that trips people up — and shows up in interviews:

The mental model to keep: diffusion never compresses — it only transports the distribution. A GAN squeezes all image variety into a hundred numbers; a VAE squeezes it into a chosen few dozen; diffusion keeps every pixel slot busy at all times and merely moves what fills those slots from "structured picture" to "static" and back.

14.7.5 The Generative Trilemma

Line up the families on three desirable axes — this trio is known as the generative trilemma:

1. Fast sampling — generate quickly, not sequentially.
2. Mode coverage — high diversity across outputs.
3. High sample quality — high inception score, low FID.

No family holds all three:

Read row by row: GAN: high-quality samples plus fast sampling, but mode coverage is generally poor — the classic mode collapse problem. VAE: fast sampling (just draw from the learned latent Gaussian and decode) plus decent mode coverage, but plain VAEs struggle to produce high-quality images. Hierarchical VAEs push quality up, and discrete VAEs such as VQ-VAE help when generating class-specific data. Diffusion: great quality and high diversity, but slow sampling — good quality demands many steps.

Add a fourth axis beyond the trilemma: training stability. Diffusion training — learning to remove noise — is a stable, well-behaved optimization. VAE training is stable too. GANs are the fragile ones: getting the loss to descend smoothly, asymptotically, is notoriously difficult there. Stability does not affect the generated output directly; it affects whether the training process behaves while you create the model. That distinction matters in practice: an unstable run may never deliver any output worth scoring, however good its theoretical ceiling.

Real-world connection: production teams pick families by constraint, not fashion — game studios needing thousands of texture variants per second lean on GAN-style one-shot generation despite tuning pain, while advertising pipelines that can wait seconds per asset choose diffusion for its diversity guarantees; VQ-VAE-style discrete latents power token-based generators where class control matters most.

14.8 Conditional Generation and Latent Diffusion

14.8.1 Conditioning on Text

Everything above generates unconditionally. For text-to-image generation, encode the text document into an embedding and inject that embedding as a condition into the denoising network at every step — conditional diffusion sampling. Mechanically it mirrors the time signal you already know: just as a sinusoidal vector tells each U-Net block when in the schedule it is working, a text-embedding vector tells each block what is being built. The two vectors travel together through all $T$ steps.

Start from several different draws of the same Gaussian with the same prompt and you get different images, because the noise samples themselves differ. The prompt fixes the destination region of the data distribution; the starting noise picks which point inside that region you visit. Same sentence, ten seeds, ten distinct pictures — all matching the words. And since outputs are color images, the process starts from three channels, not one.

14.8.2 Latent Diffusion and Stable Diffusion

Every diffusion model discussed so far ran on the original image. Latent diffusion, as the name suggests, first encodes the original image into a latent space much smaller than the pixel space. It then runs the entire diffusion process there, and a decoder maps the result back to images. Concretely, a picture on the order of $1000\times 600$ pixels might live in a latent grid on the order of $40\times 60$ — treat both sizes as illustrative magnitudes rather than exact specifications; the point is that each side shrinks by more than an order of magnitude.

Why does this help so much? Count the work. The pixel image holds about $1000\times 600=600,000$ spatial positions; the latent grid holds about $40\times 60=2,400$. Every convolution sweeps over positions and every self-attention layer compares pairs of positions, so shrinking the grid by tens of times per side slashes computation per step by similar orders — and since generation runs that cost once per timestep, the savings multiply across the whole chain. Doing the whole denoising conversation in that small space therefore cuts computation sharply while improving both generation speed and output quality — an excellent trade-off between performance and computational cost. Quality improves too because the autoencoder has already stripped out imperceptible high-frequency detail, letting the diffusion network spend its capacity on structure that actually matters. The design is also flexible: swap or adjust the autoencoder input to serve different tasks.

Real-world: state-of-the-art text-to-image systems build on this recipe — named examples included DALL-E and Meta's models. Stable Diffusion performs diffusion in the latent space of pretrained autoencoders, cutting computational load significantly without compromising performance — which is why a consumer GPU can run it at home. The same machinery extends to video: generate a sequence of frames from a text prompt, following the exact pattern used for images — a topic reserved for the closing session.

Real-world connection: open-source image tools owe their accessibility to latent diffusion — compressing before diffusing is what lets hobbyists synthesize images on laptops instead of data-center clusters, and the same trick now powers on-device photo editors that upscale, inpaint, and restyle pictures without ever uploading them.

14.9 Energy-Based Models: Modeling Probabilities from First Principles

14.9.1 Why Another Family?

Energy-based models trace back to principles of statistical mechanics — the physics discipline that describes how particle states distribute across energies. The physics stays in the background here; what matters is what the framework offers generative modeling. Every family studied so far estimates probabilities, usually after committing, explicitly or implicitly, to a parametric form: Gaussian models, mixture of Gaussians, and friends. An energy-based model hands you an extremely flexible way of defining probabilities from first principles.

Intuition + analogy. Think of a landscape with hills and valleys scattered freely across a map, and imagine sprinkling sand over it from far above: sand settles thickly in valleys, thinly on peaks, nowhere at all beyond the map's edge. If you now read the sand height at each location as an unnormalized probability, the landscape itself becomes a distribution — no formula template demanded. Energy-based models sculpt exactly such landscapes with neural networks, then convert heights into honest probabilities. Where the analogy strains: real sand piles up in one configuration; probabilities need careful total-mass bookkeeping, which is precisely the next topic.

14.9.2 What Makes a Function a Valid Density?

The doorway question, put to the room:

The continuity guess deserves its moment of respect — it sounds plausible because every density we had met was smooth. But plenty of continuous functions integrate to two, or minus three, or infinity; and plenty of perfectly legal densities have jumps (the uniform density on $[0,1]$ leaps from 0 to 1 at both ends). Continuity is not required, and having it would not help anyway. The misconception dissolves once you see what probability actually uses about a density: only areas. Negative values would let areas cancel, breaking "probabilities live in $[0,1]$", and a total area other than one breaks the guarantee that some outcome occurs. Nothing else can go wrong — so nothing else is required.

Once only those two rules constrain you, the space of candidate distributions explodes — you are free to consider almost anything.

14.9.3 Manufacturing Valid Densities

Nonnegativity is cheap to guarantee:

• Square any function: squares cannot go negative.
• Exponentiate any function: $e^{g(\theta ,x)}>0$ always, whatever sign $g$ takes.
• Take a modulus: absolute values are nonnegative by definition.

Take any nonnegative candidate $f(\theta ,x)$ and divide by its own area to force the second condition:

$$\tilde{p}(x)=\frac{f(\theta ,x{)}^2}{{\int }_{-\infty }^{\infty }f(\theta ,x{)}^{2}dx}$$

Here $\theta$ collects the tunable parameters and the denominator — a single positive number, assuming the integral exists and is finite — rescales whatever numerator you chose so the total area becomes exactly one. Verify directly:

$$\begin{array}{rlr}{\int }_{-\infty }^{\infty }\tilde{p}(x)dx& ={\int }_{-\infty }^{\infty }\frac{f(\theta ,x{)}^2}{{\int }_{-\infty }^{\infty }f(\theta ,u{)}^{2}du}dx& \text{(substitute the quotient)}\\ & =\frac{1}{\int f(\theta ,u{)}^{2}du}{\int }_{-\infty }^{\infty }f(\theta ,x{)}^{2}dx& \text{(denominator does not depend on }x\text{)}\\ & =1& \text{(numerator cancels denominator)}\end{array}$$

This quotient is always a valid density — nonnegative above, unit area below. The catch is feasibility: sampling from it, and even evaluating that integral, may not be doable unless the normalizing constant has a closed form. Classic families earn their place by having one:

• Gaussian kernel: $e^{-\frac{(x-\mu {)}^2}{2{\sigma }^2}}$ is always positive, and its integral equals $\sqrt{2\pi {\sigma }^2}$ in one dimension — the familiar normalizing constant, which is why a proper Gaussian carries the factor $\frac{1}{\sqrt{2\pi {\sigma }^2}}$ out front.
• Exponential distribution: ${\int }_0^{\infty }{e}^{-\lambda x}dx=\frac{1}{\lambda }$ (antiderivative $-\frac{1}{\lambda }{e}^{-\lambda x}$, which vanishes at $\infty$ and equals $-\frac{1}{\lambda }$ at 0), so the normalized density $\lambda e^{-\lambda x}$ on $[0,\infty )$ integrates to one.

Any parameterized family whose area can be integrated analytically into a constant works as a generative family. You can also chain several valid densities together to sculpt more complex ones, provided each ingredient individually satisfies the two rules. And once a density is parameterized, the likelihood principle applies — fit $\theta$ by maximizing likelihood as always.

14.9.4 The Extra Condition for Generation

Using a density as a generative model adds one demand beyond the two validity rules: sampling from it should be simple. A perfectly valid density you cannot draw from is useless for synthesis — knowing the exact shape of a mountain range is worthless if you cannot throw a dart that lands somewhere on it in proportion to sand depth. Extreme flexibility plus easy sampling — that combination is the whole promise of energy-based models.

Real-world connection: this normalize-by-the-area trick underlies density estimation in anomaly monitoring — industrial inspection systems score sensor readings under flexible learned densities where no Gaussian assumption fits, and the same two-rules-first mindset governs how Bayesian statisticians define priors on unusual parameters like correlation matrices.

14.10 The Energy-Based Model Formulation

14.10.1 Definition, Partition Function, Energy

The energy-based form wraps any scored function into a probability:

$$p_{\theta }(x)=\frac{\exp (F_{\theta }(x))}{Z_{\theta }},Z_{\theta }=\int \exp (F_{\theta }(x))dx$$

Here $\theta$ are the learnable parameters, $F_{\theta }(x)$ is the scorer (any real-valued function of the input), and $Z_{\theta }$ is a scalar normalizing constant; it guarantees $p_{\theta }(x)$ stays between 0 and 1. The numerator alone only promises positivity — exponentials never go negative — so this is exactly the two-rules recipe of the previous section, with exponentiation supplying rule one and division by $Z_{\theta }$ supplying rule two.

$Z_{\theta }$ is none other than the partition function from statistical physics. There it tracks how particle states distribute over energies. The negated scorer $-F_{\theta }(x)$ carries a special name: the energy, written $E(x)=-F_{\theta }(x)$. In that language the same law reads $p_{\theta }(x)\propto e^{-E(x)}$: low-energy configurations are common, high-energy ones rare. The reading is intuitive: configurations with low energy — meaning high $F_{\theta }(x)$ — are more likely, since the probability grows exactly when $F_{\theta }$ is large. Push the intuition to the extremes: if one configuration hoards all the probability, you get a fully deterministic outcome — its probability one, everything else zero; if $n$ outcomes share equally, each carries probability $1/n$. Working with log probability is natural here because the logarithm shrinks the range of very large numbers to a manageable scale — taking logs turns $p_{\theta }(x)=e^F/Z$ into $\log p_{\theta }(x)=F_{\theta }(x)-\log Z_{\theta }$, where products became sums.

14.10.2 Strengths and Difficulties

The strength: extreme flexibility — pretty much any function $F_{\theta }(x)$ you like, including an arbitrary neural network. No invertibility constraints like flows, no Gaussian shapes like classic density estimation. The difficulties come in threes:

1. Sampling is hard. Easy only when the integral admits an analytic solution — otherwise drawing samples from $p_{\theta }$ requires iterative approximation schemes.
2. Evaluating and optimizing the likelihood is hard, because computing $Z_{\theta }$ numerically scales exponentially with dimension. For a discrete variable with $d$ binary pixels there are $2^d$ states to sum over: at $d=100$, that is already about $10^{30}$ terms — no computer enumerates them. With images — hundreds of dimensions, even into the thousands, and truly huge at 4K scale — that integral becomes hopeless.
3. Feature learning is limited, though adding latent variables can help.

Hold those two hard problems down and you gain a versatile tool able to model very complex data reasonably well.

14.10.3 Compare, Do Not Evaluate

Here is the escape hatch. The partition function is a constant — the same constant — inside two probabilities you want to compare, so it cancels:

$$\begin{array}{rlr}\frac{p_{\theta }(x)}{p_{\theta }(x^{\prime })}& =\frac{\exp (F_{\theta }(x))/Z_{\theta }}{\exp (F_{\theta }(x^{\prime }))/Z_{\theta }}& \text{(write both probabilities out)}\\ & =\exp (F_{\theta }(x))\cdot \frac{Z_{\theta }}{Z_{\theta }}\cdot \exp (-F_{\theta }(x^{\prime }))& \text{(rearrange factors)}\\ & =\exp (F_{\theta }(x)-F_{\theta }(x^{\prime }))& \text{(cancel }{Z}_{\theta }\text{)}\end{array}$$

No $Z_{\theta }$ anywhere. Individual probabilities may be out of reach, yet ranking two candidates costs almost nothing — just evaluate the scorer twice and subtract. Many practical tasks need exactly that — comparison rather than evaluation — and those tasks can use energy-based models directly.

A first-semester memory proves the pattern:

Notice how deep the parallel runs: Naive Bayes' skipped denominator $P(\text{attributes})$ is a sum over all attribute combinations — a partition function in disguise. You have been exploiting cancellation since your first classifier.

14.10.4 Tasks Built on Comparison

Real-world: anomaly detection — compare the probability a trained model assigns to a real sample against a test sample. If the ratio lands much greater than one, the test point is an anomaly. A ratio near one (or lower) says the test point behaves like genuine data.

Real-world: object recognition and classification — given five candidate classes, compare the joint probability of the image with each class hypothesis: cat, dog, and so on. Assign the winning label; no absolute probability required. The same comparison pattern serves sequence labeling, part-of-speech tagging (does this word belong to this tag?), and image retrieval with a trained model. Wherever relative rankings suffice, energy-based models fit.

Real-world connection: fraud and intrusion detection systems run this exact playbook — a model scores transactions or network events, and alerts fire whenever a new event's relative score collapses against the typical band; nobody ever needs the absolute probability, only the damning gap.

14.11 The Ising Model

14.11.1 Setup: Recovering Corrupted Binary Images

Among the earliest energy-based models stands the Ising model. It handles deliberately simple data: a 3 by 3 patch of an image whose pixels hold only zeros and ones — no 8-bit gray levels. Imagine the true patch $\mathbf{y}$ (the clean pixels) and a corrupted observation $\mathbf{x}$, degraded perhaps by the camera that captured it. The task: recover the true pixels from the corrupted ones using the energy-based machinery.

Two structural assumptions drive the model:

1. Local corruption. The observed value at pixel $i$ depends only on the true value at that same pixel — no cross-pixel corruption coupling. Noise flips pixels independently.
2. Neighborhood agreement. Neighboring true pixels tend to share values, because correlation in a physical scene runs high — colors rarely change abruptly except at edges, so nearby values stay close.

The second assumption carries real content: it says clean images occupy a smooth corner of all possible patches. A patch of pure noise violates neighborhood agreement everywhere; a photo of a wall satisfies it almost everywhere. That prior belief is what lets the model outvote a corrupted pixel using its neighbors' testimony.

14.11.2 Energy Form and Recovery

Write the joint probability of true and observed pixels as an exponential of summed interaction terms:

$$P(\mathbf{y},\mathbf{x})\propto \exp \left(\sum _{i}T(y_i,x_i)+\sum _{(i,j)\in \text{neighbors}}T(y_i,y_j)\right)$$

Here $i$ runs over all pixel sites and each ordered neighbor pair $(i,j)$ contributes once. The function $T(\cdot ,\cdot )$ is a potential — a score table assigning points to value combinations. The first sum encodes observation-versus-truth at every site $i$: reward when the reconstruction matches what the camera saw. The second rewards agreeing neighbors. No three-way or global terms appear — every factor touches either one site's truth-versus-observation or one pair of neighboring truths, which is precisely the "no cross terms" structure of the classical model. Because everything lives inside an exponential of sums, this is an energy-based distribution of exactly the Section 14.10 form, with the energy gathering penalty terms from mismatches and disagreements.

Recovery then asks for the most probable truth: maximize $P(\mathbf{y}\mid \mathbf{x})$. Since the observation $\mathbf{x}$ is fixed during restoration, $P(\mathbf{y}\mid \mathbf{x})=P(\mathbf{y},\mathbf{x})/P(\mathbf{x})$, and dividing by the constant $P(\mathbf{x})$ cannot reorder candidates — so equivalently maximize the joint $P(\mathbf{y},\mathbf{x})$, choosing suitable potential functions along the way. Notice the partition-function escape hatch at work: the normalizing constant never enters the argument, because argmax ignores constants. Solving that maximization returns the restored image values.

Real-world connection: the same pairwise-potential machinery powers Markov random field methods used in medical image denoising and satellite imagery cleanup — restoring binary land-cover maps where sensor dropouts must be repaired without erasing real field boundaries — and, historically, it modeled ferromagnetism, where atomic spins align with neighbors exactly like agreeing pixels.

14.12 Product of Experts

14.12.1 Many Experts, One Probability

Suppose you have trained several scoring models — call them $Q_{{\theta }_1}$, $R_{{\theta }_2}$, $T_{{\theta }_3}$. They can be anything: a flow model, a PixelCNN, even a nondifferentiable scorer. Treat each one as an expert that independently judges how likely an input $x$ is. The product of experts multiplies their judgments and normalizes:

$$p(x)=\frac{1}{Z}\prod _k{\text{expert}}_k(x)$$

Here $k$ indexes the experts, $Z=\int {\prod }_k{\text{expert}}_k(u)du$ is once again a normalizing constant, and each ${\text{expert}}_k(x)\ge 0$ is an unnormalized opinion score.

Two views make the construction transparent. In log space, taking logs turns the multiplication into addition:

$$\begin{array}{rlr}\log p(x)& =\sum _k\log {\text{expert}}_k(x)-\log Z& \text{(log of product = sum of logs)}\\ & =-\left(\sum _k{E}_k(x)\right)-\log Z& \text{(write each }-\log {\text{expert}}_k=E_k\text{)}\end{array}$$

So a product of experts is itself an energy-based model whose total energy is the sum of the individual expert energies — every expert adds its own penalty terms, and bad news accumulates additively. In set-theory language, multiplying densities carves out the intersection of what the experts consider plausible; each new expert shrinks the surviving region rather than broadening it.

The behavior is logical AND: the combined probability is high only when every expert votes confidently; if even one expert outputs zero, the whole product collapses to zero. Compare that with a mixture of Gaussians, where probabilities are summed — an OR-flavored combination: one enthusiastic expert suffices to lift a mixture's mass at some point, while a product demands unanimity. Product of experts has powered varied image-generation variants precisely because AND-composition sculpts sharp, specific requirements.

When to pick which: multiply when requirements must co-occur; mix when alternatives are acceptable.

14.12.2 Attribute-Controlled Faces

Real-world: face generation by attribute composition. Take an expert for "young", an expert for "female", an expert for "smiling" — multiply them and sample. The outputs are young, female, smiling faces, because all three expert scores must be simultaneously high. Swap in an expert for wavy hair and the attribute list grows arbitrarily. Complex, controllable generation falls out of multiplying simple opinions.

Real-world connection: attribute-composed face synthesis anticipated today's controllable generation APIs — stock-image tools let designers request "smiling woman in her twenties with curly hair" by combining attribute models multiplicatively, and the same AND-composition idea reappears in classifier-guided diffusion, where a guidance term pushes every denoising step toward the requested condition.

Exam Guidance Summary

• Expect assignment-related questions in the final exam. Assignments double as exam preparation material, so finish yours thoroughly — the concepts you implemented are the concepts most likely to reappear on paper.
• Assignment marks reserve points for tuned runs producing reasonable results — submitting code that merely runs without sensible output leaves marks on the table. Treat tuning as part of the graded task: adjust learning rates, schedules, and training time until samples actually look sensible.
• The latent-dimension comparison across diffusion, VAE, and GAN is a favorite short-interview question; know it cold. One-line version: GAN and VAE codes are compact (roughly 100 numbers for a 128×128 image), while diffusion's noise draw matches the full image resolution.
• Be ready to explain the generative trilemma: which family delivers fast sampling, which delivers diversity, which delivers quality, and why training stability separates diffusion and VAE from GAN. Practice naming which corner each family sacrifices and why.
• Know which metric rewards diversity (FID, lower is better) and which rewards quality (IS, higher is better); expect a short conceptual question on them.
• Understand why DDIM exists: the speed motivation and the data-prediction trick that permits skipping steps. Being able to write ${\hat{\mathbf{x}}}_{\theta }({\mathbf{x}}_t,t)=({\mathbf{x}}_t-\sqrt{1-{\bar{\alpha }}_t}{\mathbf{ϵ}}_{\theta })/\sqrt{{\bar{\alpha }}_t}$ from the forward closed form is the core skill here.

A practical revision order: first memorize the three forward-process constants (${\beta }_t$, ${\alpha }_t$, ${\bar{\alpha }}_t$) and the closed form built from them; then rehearse the loss-collapse story (ELBO → Gaussian KLs → squared noise error); finally drill the comparison tables — families versus trilemma corners, FID versus IS — since those map directly onto short-answer questions.

Key Industry Applications

• Conversational image editing: tools in the ChatGPT ecosystem modify uploaded images through diffusion-like reverse processes, which is why their generation feels slow — iterative denoising is inherent. DDIM-style step-skipping is the standard remedy shipping in production samplers.
• Biomedical imaging: U-Net originated in biomedical image segmentation, assigning every pixel a segment label; the same architecture now powers diffusion noise prediction — one design serving both clinical contouring tools and creative generators.
• Text-to-image systems: DALL-E and Meta's generators build on latent diffusion, and Stable Diffusion runs diffusion in the latent space of pretrained autoencoders for speed without quality loss — the reason consumer GPUs can synthesize images at home.
• Text-to-video: sequences of frames generated from text prompts extend latent diffusion frame by frame, turning the image recipe into a temporal one.
• Anomaly detection: energy-based comparison of sample probabilities flags outliers in industrial machine monitoring and fraud screening — the ratio test needs no partition function at all.
• Recognition, tagging, retrieval: classification among candidate classes, part-of-speech tagging, sequence labeling, and image retrieval all reduce to probability comparisons suited to energy-based models — wherever ranking suffices, absolute probabilities can stay uncomputed.
• Attribute-controlled media: product-of-experts compositions generate faces meeting simultaneous attribute demands — young, female, smiling, wavy-haired — by construction; the AND-composition idea echoes in today's guided-diffusion conditioning tricks.